Over eight ago I wrote a post ZFS is not always the answer. Bring back gmirror!, suggesting that writing off UFS in favour of ZFS wasn’t a clear cut decision and reminding people how gmirror could be used to mirror drives is you needed redundancy. It’s still true, but it probably needs an update as things are done a little differently now.
MBR vs GPT
There have been various disk partition formats over the years. The original PDP-11Unix contained only a boot block (512b) to kick start the OS, but BSD implemented its own partitioning scheme from 386BSD onwards – 8K long consisting of a tiny boot1 section that was just enough to find boot2 in the same slice, which was then able to read UFS and therefore the kernel. This appeared 4.2BSD on the VAX.
Then from the early 1990s the “standard” partition scheme from the MS-DOS Master Boot Record (MBR) seemed like a great idea. Slices got replaced by partitions and you could co-exist with other systems on the same drive, and x86 systems were now really common.
The so-called MBR scheme had its problems (and workarounds) as Microsoft wasn’t exactly thinking ahead but these have been fixed with the wonderful GPT scheme, which was actually designed. However, GEOM Mirror and UFS predate GPT adoption and you have to be aware of a few things if you’re going to use them together. And you should be using GPT.
Why should you use GPT just because it’s “new”? It was actually dreamt up more than 25 years by Intel (on the IA-64 I believe). It has a backup header so if you lose the first blocks on your drive you’re not dead in the water – a favourite trick with DOS/Windows losing the entire drive for the sake of one sector. GPT allows drives to be more than 2Tb because it has 64-bit logical block addresses. If that’s not enough, it identifies partitions with a UUID so you can move them around physically without having problems, and if you’re mixing operating systems on the same disk the others are likely to be using GPT too, so they’ll play nice. As long as you have UEFI compatible firmware, you’re good to go. If all your drives are <2TB and you have old firmware, and only want to run FreeBSD, stick to MBR – and keep a backup of the boot block on a floppy just in case.
Gmirror and GPT
As I mentioned, GPT keeps a second copy of the partition information on the disk. In fact it stores a copy at the end of the drive, and if the one at the front is corrupt or unreadable it’ll use that instead. Specifically GPT stores header in LBA 1 and the partition table in LBA 2-33 (insanely large partition table but Intel didn’t want to be accused of making the same limiting mistakes as Microsoft).
The backup GPT header is on on the last block of the drive, and the partition table going backwards from that (for 33 LBAs).
GMirror, meanwhile, stores its metadata on the last 512-byte sector of the drive. CRUNCH.
So what to do? One method is to use the -h switch when setting up with gmirror:
gmirror label -h m0 da0 da1This moves the metadata to the front of the disk, which will deconflict it with the GPT header okay but might crunch with other bootloaders, particularly from another OS that’s sharing the same disk, and which we have no control of. I say might. Personally, I wouldn’t be inclined to take the risk unless I’m dedicating the drive to FreeBSD.
The safe method is to NOT mirror the entire disk, only the partitions we’re interested in. Conventionally, and in the 2017 post, you mirrored the entire drive and therefore the drives were functionally identical without any further work. The downside was that if you replaced a drive you needed one exactly the same size (or larger), and not all 500Gb drives are the same number of blocks (although there’s a pretty good chances these days).
GEOMs and disks?
I’ve explained how to mirror a single partition already, but not gone into the technicalities. If you’re new to FreeBSD you might not have cottoned on what a GEOM is. It’s short for “geometry”, which probably doesn’t help with understanding it one bit.
It gets the name from disk geometry, but don’t worry about the name. It’s an abstraction layer added to FreeBSD between the physical drive (provider) and higher level functions of the OS such as filing systems (consumers). You can add GEOM classes between the provider and consumer to provide RAID, mirroring, encryption, journaling, volume management and suchlike. Before ZFS, this was how you got fancy stuff done. Now, not so much. But the GEOM mirror class (aka gmirror) is still very useful indeed.
But the bottom line is that a disk partition can be a provider in just the same way as the whole disk, so what works for a disk will also work for a partition. Chances are the installer has partitioned up your drive thus:
=> 40 5860533088 ada0 GPT (2.7T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 4194304 2 freebsd-swap (2.0G)
4196352 5856335872 3 freebsd-ufs (2.7T)
5860532224 904 - free - (452K)
This means /dev/ada0p3 is the UFS partition we’re interested in mirroring. Believe it or not, partition number starts a one, not zero!
How to actually do it
So if you’ve installed your system and now want to add a GEOM mirror, proceed as follows. Let’s assume your second drive is ada1, which would be logical.
You’ll have to partition it so it has at least one partition the same size as the one you want to mirror. Chances are you’ll want all partitions. The quickest way to achieve this is to simply copy the partition table:
gpart backup ada0 | gpart restore -F ada1You can sanity check this with gpart show ada1, which should be the same as gpart show ada0.
Load the geom_mirror module
kldload geom_mirror
echo 'geom_mirror_load="YES"' >> /boot/loader.confThe second line adds it to loader.conf to make it load each time, but only do it if it’s not there already. The kldload will complain if it’s already loaded, which is a good clue you don’t need the second line.
Create the mirror
gmirror label ufsroot /dev/ada0p3 /dev/ada1p3The “label” subcommand simply writes the metadata to the disks or partitions – remember they’re all the same to GEOM. The name “ufsroot” is chosen by me to be meaningful. Manuals use things like gm0 for GEOM mirrors and people have come to think it’s important they’re named this way, when the opposite is true. You already know it’s a GEOM mirror because the device is in /dev/mirror – it’s more helpful to know what it’s used for, e.g. UFS root, or swap, or var or whatever.
You can, while you’re at it, mirror as many partitions as you wish if you have separate ones for other purposes. You can even mirror a zfs partition without it knowing if you’re crazy enough. Mirroring the swap partitions is something you should definitely consider.
You can check it’s worked with gmirror status, which should output something like this:
Name Status Components
mirror/ufsroot COMPLETE ada0p3 (ACTIVE)
ada1p3 (SYNCHRONIZING)Wait until it’s finished synchronising, which will take a long time on a large disk. Perhaps go to bed.
Mount the mirror
This process will have created a new device called /dev/mirror/ufsroot but you still have to mount it in place of the “old” UFS partition. This is controlled in the normal way by /etc/fstab, so make a backup and fire up your favourite editor.
Look for the entry for /dev/ada0p3 and change it to /dev/mirror/ufsroot:
/dev/mirror/gm0 / ufs rw 1 1Reboot and you should be good.
Boot code
Although your UFS partition is mirrored, if ada0 fails now the system won’t boot as ada1 lacks the boot code. You can add this this easily enough:
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1Finally, what about swap partitions? For robustness, mirror them too in the same way:
gmirror label swap /dev/ada0p2 /dev/ada1p2Then edit fstab to swap on /dev/mirror/swap
Alternatively you can edit fstab to swap on ada1p2 as well (best for performance). Or you can just leave it as it is – if ada0 fails and you reboot you’ll have no swap until you fix it, but you’ll probably be worrying about other things.