(for the rest of us)
I don’t know about you but most of my ZFS arrays are large, using SAS drives connected via SAS HBAs
to expanders that know which disk is where. I also have multiple redundancy in the zpool and hot spares, so I don’t need pay a visit just to replace a failed disk. And if I do I can get the enclosure to flash an LED over the drive I’m interested in replacing.
Except at home. At home I’ve got what a lot of people probably have. A small box with a four-drive cage running RAIDZ1 (3+1). And it’s SATA, because it really is a redundant array of independent drives. I do, of course, keep a cold spare I can swap out. Always make sure you you have at least one spare drive of the dimensions of a RAIDZ group, and know where you find it.
And to make it even more fun, you’re booting from the array itself.
After many years I started getting an intermittent CAM error, which isn’t good news. Either one of the drives was loose, or it was failing. And there’s no SAS infrastructure to help. If you’re in a similar position you’ve come to the right place.
WARNING. The examples in this article assume ada1 is the drive that’s failed. Don’t blindly copy/paste into a live system without changing this as appropriate
To change a failed or failing drive:
- Find the drive
- Remove the old drive
- Configure the new drive
- Tell the RAIDZ to use it
Finding the failed drive
First, identify your failing drive. The console message will probably tell you which one. ZFS won’t, unless it’s failed to the extent it’s been offlined. “zpool status” may tell you everything’s okay, but the console may be telling you:
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): CAM status: ATA Status Error
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): CAM status: ATA Status Error
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): RES: 41 40 b0 71 20 00 f6 00 00 00 01
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): Retrying command, 0 more tries remain
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 70 20 40 f6 00 00 01 00 00
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): CAM status: ATA Status Error
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): RES: 41 40 b0 71 20 00 f6 00 00 00 01
Feb 2 17:56:23 zfs1 kernel: (ada1:ahcich1:0:0:0): Error 5, Retries exhausted
So this tells you that ada1 is in a bad way. But which is ada1? It might be the second one across in your enclosure, or it might not. You’ll need to do some hunting to identify it positively.
Luckily most disks have their serial number printed on the label, and this is available to the host. So finding the serial number for ada1 and matching it to the disk label is the best way – if you’ve only got four drives to check, anyway.
I know of at least five ways to get a disk serial number in FreeBSD, and I’ll list them all in case one stops working:
dmesg
Just grep for the drive name you’re interested in (ada1). This is probably a good idea as it may give you more information about the failure. If FreeBSD can get the serial number it will display it as it enumerates the drive.
geom disk list
This will print out information on all the geoms (i.e. drives), including the serial number as “ident”
camcontrol identify ada1
This gives you more information that you ever wanted about a particular drive. This does include the serial number.
diskinfo -s /dev/ada1
This simply prints the ident for the drive in question. You can specify multiple arguments so diskinfo -s /dev/ada? works (up to ten drives).
smartctl -i /dev/ada1
Smartctl is a utility for managing SATA drives (not SAS!), and you should probably install it. It’s part of Smartmontools as it gives you the ATA information for drive, including errors rates, current temperature and suchlike.
Whichever method works for you, once you’ve got your serial number you can identify the drive. Except, of course, if your drive is completely fubared. In that case get the serial numbers of the drives that aren’t and identify it by elimination.
Saving the partition table from the failed drive.
In readiness for replacing it, if you can save its partition table:
gpart backup ada1 > gpart.ada1
If you can’t read it, just save another one from a different drive in the vdev set – they should be identical, right?
Swapping the bad drive out.
Next, pull your faulty drive and replace it with a new one. You might want to turn the power off, although it’s not necessary. However, it’s probably safer to reboot as we’re messing with the boot array.
Try zpool status, and you’ll see something like this:
pool: zrpool: zr
state: DEGRADED
status: One or more devices could not be opened.
Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
scan: scrub in progress since Sun Feb 2 17:42:38 2025
356G scanned at 219/s, 175G issued at 108/s, 10.4T total
0 repaired, 1.65% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUMzr DEGRADED 0 0 0raidz1-0 DEGRADED 0 0 0ada0p3 ONLINE 0 0 016639665213947936 UNAVAIL 0 0 0 was /dev/ada1p3ada2p3 ONLINE 0 0 0ada3p3 ONLINE 0 0 0
It’s complaining because it can’t find the drive with the identity 16639665213947936. ZFS doesn’t care where the drives in a vdev are plugged in, only that they exist somewhere. Device ada1 is ignored – it’s just got some random disk in that zfs isn’t interested in.
Setting up the replacement drive
So let’s get things ready to insert the new drive in the RAIDZ.
First restore its partition table:
gpart restore /dev/ada1 < gpart.ada1
If you see “gpart: geom ‘ada1’: File exists”, just run “gpart destroy -F ada1”. Without the -F it may say the drive is in use, which we know it isn’t.
Next, if you’ve got a scrub going on, stop it with “zpool scrub -s zr”
As a sanity check, run “gpart show” and you should see four identical drives.
Boot sector and insertion
Now this is a boot from ZFS situation, common on a home server but not a big one. The guides from Solaris won’t tell you about this step. To make sure the system boots you need to have the boot sector on every drive (ideally). Do this with:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
And Finally… tell ZFS to insert the new drive:
zpool replace ada1p3
Run “zpool status” and you’ll see it working:
pool: zr
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Feb 2 18:44:58 2025
1.83T scanned at 429M/s, 1.60T issued at 375M/s, 10.4T total
392G resilvered, 15.45% done, 0 days 06:48:34 to go
config:
NAME STATE READ WRITE CKSUMzr DEGRADED 0 0 0raidz1-0 DEGRADED 0 0 0ada0p3 ONLINE 0 0 0replacing-1 UNAVAIL 0 0 016639665213947936 UNAVAIL 0 0 0 was /dev/ada1p3/oldada1p3 ONLINE 0 0 0ada2p3 ONLINE 0 0 0ada3p3 ONLINE 0 0 0
errors: No known data errors
It’ll chug along in the background re-silvering the whole thing. You can carry on using the system, but it’s performance may be degraded until it’s done. Take a look at the console to make sure there are no CAM errors indicating that the problem wasn’t the drive at all, and go to bed.
If you reboot or have a power cut while its rebuilding it will start from scratch, so try to avoid both!
In the morning, zpool status will return to this, and all will be well in the world. But don’t forget to order another cold spare so you’re ready when it happens again.
pool: zr
state: ONLINE
scan: resilvered 2.47T in 0 days 11:52:45 with 0 errors on Mon Feb 3 06:37:43 2025
config:
NAME STATE READ WRITE CKSUMzr ONLINE 0 0 0raidz1-0 ONLINE 0 0 0ada0p3 ONLINE 0 0 0ada1p3 ONLINE 0 0 0ada2p3 ONLINE 0 0 0ada3p3 ONLINE 0 0 0
errors: No known data errors
As a final tip, if you use diskinfo -v adaX it will tell you the type of drive and other information, which is really handy if you’re ordering another cold spare.
# diskinfo -v adaa
ada2
512 # sectorsize
3000592982016 # mediasize in bytes (2.7T)
5860533168 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
5814021 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
HGST HUS724030ALE640 # Disk descr. <- Drive to order on eBay!
PK2234P9GWJ42Y # Disk ident.
No # TRIM/UNMAP support
7200 # Rotation rate in RPM
Not_Zoned # Zone Mode