Mirroring is a technology used by many corporations and home users to back up data without interruption. When a mirror exists, it simply means that diskB replicates diskA. Or, perhaps diskC+D replicates diskA+B. Regardless of the disk configuration, the important aspect is that information on one disk or partition is being replicated. Later, that information could be more easily restored, backed up without causing service or access interruption, and even be physically stored in a data safe.
In this example, FreeBSD has already been installed on a single disk, ada0. A new disk, ada1, has been added to the system. A new mirror will be created first on the new single disk, the existing system copied to it, and then the mirror will be configured to use the both disks. This slightly complex procedure is needed because the mirror configuration needs a 512-byte information at the end of each disk though the ada0 uses the whole disk in most cases. Partitions on the new mirror can be larger or smaller than those on the existing disk, but they must be large enough to hold the data already present on ada0.
It is traditional and often desirable for the two mirror drives to be identical in model and capacity, but it is not required. Mirrors created with dissimilar drives will have a capacity equal to that of the smallest drive in the mirror. Drives inserted into the mirror later must have at least as much capacity as the smallest drive already in the mirror.
Warning: The mirroring procedure shown here is non-destructive, but as with any major disk operation, a complete backup should be completed and verified first.
Note: MBR partitioning is used here because full disks with GPT partition tables cannot be mirrored with gmirror(8). In that situation, both GPT and gmirror(8) store metadata at the end of the disk, and one will overwrite the other. It is possible to instead mirror individual GPT partitions. gmirror(8) will keep metadata inside each partition, and will not interfere with the GPT metadata at the end of the disk. That technique will not be covered here, but it is not much more complex than the MBR setup shown.
gmirror(8) requires a kernel module, geom_mirror.ko, either built into the kernel or loaded at boot- or run-time. Manually load the kernel modules now:
# gmirror load
If any mirror or label metadata, or partition table already exists on the new drives, it can be erased with glabel(8), gmirror(8), or dd(1).
# glabel clear -v /dev/ada1 # gmirror clear -v /dev/ada1 # dd if=/dev/zero of=/dev/ada1 bs=512 count=1024
Device names and numbers vary depending on where and how the drive is connected. ada0 may become ada6 if it is plugged into a different port, or the name may change from ada to da if it is moved from a SATA port to a USB SATA adapter.
First, check the media size of the primary disk by using gpart(8).
# gpart list ada0 | grep size | tail -2 Mediasize: 1000204821504 (931G) Sectorsize: 512
Create a mirror on the new single disk. The new disk must be equal to or larger than the above media size.
# geom zero load # gnop -s 1000204821504 gzero # gmirror label -v gm0 gzero.nop ada1 # gmirror forget gm0
As the result of the above commands, mirror/gm0 with
the same size as ada0 is created by using ada1. Note that the -s
1000204821504 in the second line must be equal to the media size of ada0 shown in the output of gpart(8).
The technical details are as follows. In the above commands, the kernel module geom_zero.ko is loaded, a pseudo block device /dev/gzero.nop which has the same size as ada0 is created, and a mirror mirror/gm0 is configured by using /dev/gzero.nop and ada1. The /dev/gzero.nop is not a real block device and just used to limit the capacity of mirror/gm0 (gmirror(8) uses the smallest one in the member disks as the capacity of mirror/gm0). The fourth command removes the pseudo device from mirror/gm0.
Other than the device name, mirror/gm0 acts just like a single drive. MBR and bsdlabel partition tables can now be created on the new mirror with gpart(8). This example shows a traditional split-filesystem layout, but a single / filesystem and a swap partition will work just as well.
After creating mirror/gm0, check the partition table on ada0.
# gpart show ada0
=> 63 1953525104 ada0 MBR (931G)
63 1953525042 1 freebsd [active] (931G)
1953525105 62 - free - (31k)
This output is an example of a 1 TB drive. If there is some free part at the last of the drive, it is possible to copy the contents from ada0 with an additional 512-byte information at the end of the disk.
However, if the output is something like the following, the whole disk is already allocated for FreeBSD. It means there is no space to store the 512-byte information to configure the mirror.
# gpart show ada0
=> 63 1953525105 ada0 MBR (931G)
63 1953525105 1 freebsd [active] (931G)
In this case, edit the partition table and reduce the capacity by one sector on mirror/gm0. The detail procedure will be explained later.
In either case, partition tables on the primary disk should be copied first. It can be done by using gpart(8) backup and restore subcommands.
# gpart backup ada0 > table.ada0 # gpart backup ada0s1 > table.ada0s1
The above commands will create two files, table.ada0 and table.ada0s1. The following is an example of a 1 TB drive.
# cat table.ada0 MBR 4 1 freebsd 63 1953525105 [active]
# cat table.ada0s1 BSD 8 1 freebsd-ufs 0 4194304 2 freebsd-swap 4194304 33554432 4 freebsd-ufs 37748736 50331648 5 freebsd-ufs 88080384 41943040 6 freebsd-ufs 130023424 838860800 7 freebsd-ufs 968884224 984640881
If the whole disk was used in the output of gpart(8) show, the capacity in these partition tables must be reduced by one sector. Edit the two files like the following.
# cat table.ada0
MBR 4
1 freebsd 63 1953525104 [active]
# cat table.ada0s1
BSD 8
1 freebsd-ufs 0 4194304
2 freebsd-swap 4194304 33554432
4 freebsd-ufs 37748736 50331648
5 freebsd-ufs 88080384 41943040
6 freebsd-ufs 130023424 838860800
7 freebsd-ufs 968884224 984640880
If not, these two files can be used without modification.
Now restore the partition table into mirror/gm0.
# gpart restore mirror/gm0 < table.ada0 # gpart restore mirror/gm0s1 < table.ada0s1
Check the partition table by using gpart(8) show. This example has gm0s1a for /, gm0s1d for /var, gm0s1e for /usr, gm0s1f for /data1, and gm0s1g for /data2.
# gpart show mirror/gm0
=> 63 1953525104 mirror/gm0 MBR (931G)
63 1953525042 1 freebsd [active] (931G)
1953525105 62 - free - (31k)
# gpart show mirror/gm0s1
=> 0 1953525042 mirror/gm0s1 BSD (931G)
0 2097152 1 freebsd-ufs (1.0G)
2097152 16777216 2 freebsd-swap (8.0G)
18874368 41943040 4 freebsd-ufs (20G)
60817408 20971520 5 freebsd-ufs (10G)
81788928 629145600 6 freebsd-ufs (300G)
710934528 1242590514 7 freebsd-ufs (592G)
1953525042 63 - free - (31k)
Two of the partition tables should have some free part at the end of each disk.
Then, create new filesystems on these partitions. The number of partitions depends on the primary disk.
# newfs -U /dev/mirror/gm0s1a # newfs -U /dev/mirror/gm0s1d # newfs -U /dev/mirror/gm0s1e # newfs -U /dev/mirror/gm0s1f # newfs -U /dev/mirror/gm0s1g
Make the mirror bootable by installing bootcode in the MBR and bsdlabel and setting the active slice:
# gpart bootcode -b /boot/mbr mirror/gm0 # gpart set -a active -i 1 mirror/gm0 # gpart bootcode -b /boot/boot mirror/gm0s1
Adjust the /etc/fstab to use the new partitions on the mirror. Back up this file first by copying it to /etc/fstab.orig.
# cp /etc/fstab /etc/fstab.orig
Then edit /etc/fstab. Replace the all of /dev/ada0 with mirror/gm0.
# Device Mountpoint FStype Options Dump Pass# /dev/mirror/gm0s1a / ufs rw 1 1 /dev/mirror/gm0s1b none swap sw 0 0 /dev/mirror/gm0s1d /var ufs rw 2 2 /dev/mirror/gm0s1e /usr ufs rw 2 2 /dev/mirror/gm0s1f /data1 ufs rw 2 2 /dev/mirror/gm0s1g /data2 ufs rw 2 2
Also, in case the gmirror(8) kernel module has not been built into the kernel, /boot/loader.conf is edited to load it:
geom_mirror_load="YES"
Filesystems from the original disk can now be copied onto the mirror with dump(8) and restore(8). Note that
it may take some time to create a snapshot for each filesystem by the dump(8) command with
-L option.
# mount /dev/mirror/gm0s1a /mnt # dump -C16 -b64 -0aL -f - / | (cd /mnt && restore -rf -) # mount /dev/mirror/gm0s1d /mnt/var # mount /dev/mirror/gm0s1e /mnt/usr # mount /dev/mirror/gm0s1f /mnt/data1 # mount /dev/mirror/gm0s1g /mnt/data2 # dump -C16 -b64 -0aL -f - /usr | (cd /mnt/usr && restore -rf -) # dump -C16 -b64 -0aL -f - /var | (cd /mnt/var && restore -rf -) # dump -C16 -b64 -0aL -f - /data1 | (cd /mnt/data1 && restore -rf -) # dump -C16 -b64 -0aL -f - /data2 | (cd /mnt/data2 && restore -rf -)
Reboot the system. If everything works fine, the system should boot by using mirror/gm0. The mirror/gm0 should have the same contents as the ada0 had. Just after the reboot, the system is loaded from mirror/gm0 and ada0 is no longer used. Note that the mirror still consists of ada1 only.
If the boot stopped with the following message, there was something wrong.
Mounting from ufs:/dev/mirror/gm0s1a failed with error 19.
Loader variables:
vfs.root.mountfrom=ufs:/dev/mirror/gm0s1a
vfs.root.mountfrom.options=rw
Manual root filesystem specification:
<fstype>:<device> [options]
Mount <device> using filesystem <fstype>
and with the specified (optional) option list.
eg. ufs:/dev/da0s1a
zfs:tank
cd9660:/dev/acd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)
? List valid disk boot devices
. Yield 1 second (for background tasks)
<empty line> Abort manual input
mountroot>
To recover from this, enter ufs:/dev/ada0s1a to the prompt. Although the system should boot by using ada0, another prompt to select a shell program appears because /etc/fstab is incorrect. Just hit the enter key at the prompt. It is possible to restore the modifications so far by reverting /etc/fstab. Reboot the system and try the procedure again.
Enter full pathname of shell or RETURN for /bin/sh: # cp /etc/fstab.orig /etc/fstab # reboot
If the system booted on mirror/gm0 successfully, the final step is adding ada0 as a spare disk.
Important: Adding ada0 as a spare disk means the contents on that disk will be erased completely. Double-check if mirror/gm0 has the same contents as ada0. If there is something wrong with the contents copied by dump(8) and restore(8), revert /etc/fstab, reboot, and try the whole procedure again.
# gmirror insert mirror/gm0 /dev/ada0 GEOM_MIRROR: Device gm0: rebuilding provider ada0
Synchronization between the two disks will start immediately. The gmirror(8) status command shows the progress.
# gmirror status
Name Status Components
mirror/gm0 DEGRADED ada1 (ACTIVE)
ada0 (SYNCHIRONIZING, 64%)
After a while, the synchronization will finish.
GEOM_MIRROR: Device gm0: rebuilding provider ada0 finished.
# gmirror status
Name Status Components
mirror/gm0 COMPLETE ada1 (ACTIVE)
ada0 (ACTIVE)
The mirror/gm0 now consists of the two disks, ada0 and ada1, and the contents are automatically synchronized with each other. In use, the mirror mirror/gm0 will behave just like the original single drive.
If the system boots up to a prompt similar to:
ffs_mountroot: can't find rootvp Root mount failed: 6 mountroot>
Reboot the machine using the power or reset button. At the boot menu, select option six (6). This will drop the system to a loader(8) prompt. Load the kernel module manually:
OK? load geom_mirror OK? boot
If this works then for whatever reason the module was not being loaded properly. Check whether the relevant entry in /boot/loader.conf is correct. If the problem persists, place:
options GEOM_MIRROR
in the kernel configuration file, rebuild and reinstall. That should remedy this issue.
The wonderful part about disk mirroring is that when a disk fails, it may be replaced, presumably, without losing any data.
Considering the previous RAID1 configuration, assume that da1 has failed and now needs to be replaced. To replace it, determine which disk has failed and power down the system. At this point, the disk may be swapped with a new one and the system brought back up. After the system has restarted, the following commands may be used to replace the disk:
# gmirror forget gm0
# gmirror insert gm0 /dev/da1
Use the gmirror status
command to monitor the progress of the rebuild. It is that simple.