The FreeBSD Diary

This article describes how I created a RAID-1 cluster on my FreeBSD 7.0-STABLE box using gmirror.

NOTE: when using FreeBSD 9.0, and GPT instead of MBR, I used the method describe in the comments. You may find that guide useful.

Back in March 2008, I bought an AMD Phenom(tm) and starting putting it together. You can read more about the box and the dual monitor setup here:

• Using two monitors with X.org
• Building an AMD Phenom box

• workstation
• jail server

I will use the box as a workstation for developing FreshPorts, writing articles for FreeBSD Diary, and everyday use. The box has plenty of spare cycles and extra disk space. I will put that to good use by creating several jails on it. The various jails will perform these functions:

• Regression testing for Bacula
• Downloading database backups and test loading them (FreshPorts, BSDCan, etc)

That may not sound like much, but there will be at least 8 jails on this server. Bacula regression testing will be against both MySQL and PostgreSQL and several versions of each of those databases.

Today I will be adding RAID-1 to this server. Software RAID. I will make use of the FreeBSD gmirror utility.

On this system, FreeBSD is already installed. Usually, one sets up gmirror when installing the base system. In this case, I will boot from a FreeBSD 7.0-RELEASE LIVE CD and use that to setup the RAID-1 cluster.

When setting up gmirror, you will tell it to use one HDD as the master. The other will be a slave. These are not the correct technological terms, but they will do for my purposes. The critical part of this process is making sure that you select the correct HDD for the master. If not, you will lose your installed system.

My cunning plan to correctly identify the HDD was simple. Boot from the HDD and create a distinctly unique filename in the root directory. Then boot from the live CD, mount the HDD, and find the file. Simple. What could go wrong?

The ISO image I downloaded was 7.0-RELEASE-amd64-livefs.iso. The livefs part is the key. You boot this like any other CD. From the main menu, select Fixit, then CDROM/DVD. You should then see a Fixit# prompt.

Using dmesg I found I had two HDD mounted, ad4 and ad6. Let's try mounting them and see what I can find:

Fixit# mount /dev/ad4s1 /mnt
Fixit# ls /mnt
.                               home
..                              lib
.cshrc                          libexec
.profile                        media
.snap                           mnt
COPYRIGHT                       mnt2
THIS.IS.6.3.BLOW.It.away        proc
bin                             rescue
boot                            root
cdrom                           sbin
compat                          sys
dev                             tmp
dist                            usr
entropy                         var
etc

Now let's try the other HDD and see what it contains:

Fixit# umount /mnt
Fixit# mount /dev/ad6s1 /mnt
Fixit# ls /mnt
.cshrc          boot/      entropy/     media/   sbin/
.profile        cdrom/     etc/         mnt/     sys@
.snap/          compat@    home@        proc/    tmp/
COPYRIGHT       dev/       lib/         rescue/  usr/
bin/            dist/      libexec/     root/    var/
Fixit#

I have before me, a copy of The Best of FreeBSD Basics by Dru Lavigne. I used the section Using Software RAID-1 With FreeBSD. This is not the first time I've used this documentation. I previously upgraded my gateway/firewall box to use gmirror.

The first step is to create a mirror/duplex. It is critical that you specify the drive that you wish to mirror, i.e. the master drive. In my case, as found above, that drive is /dev/ad6.

Fixit# gmirror label -v -b round-robin gm0 /dev/ad6
Metadata value stored on /dev/ad6
Done

This created gm0 and uses /dev/ad6 as the master for this mirror. At this stage, the data has not yet been mirrored. All we have done is specify that we want to mirror.

I will now mount /dev/ad6s1 so I can alter a few files.

Fixit# mount /dev/ad6s1 /mnt

The following line was added to /mnt/boot/loader.conf to ensure gmirror started at boot time:

geom_mirror_load="YES"

Instead of mounting /dev/ad6* at the next boot, we want to mount /dev/mirror/gm0. This is how FreeBSD will access your HDD, not directly, but through the mirror. This is similarly to how hardware RAID works; the RAID cards presents a device to the OS, and the OS accesses that device, e.g. twed0.

I will now alter /mnt/etc/fstab. I will change all ad6 references to mirror/gm0

Fixit# vi /mnt/etc/fstab

:%s/ad6/gmirror\/gm0/g

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/mirror/gm0s1b      none            swap    sw              0       0
/dev/mirror/gm0s1a      /               ufs     rw              1       1
/dev/mirror/gm0s1e      /tmp            ufs     rw              2       2
/dev/mirror/gm0s1f      /usr            ufs     rw              2       2
/dev/mirror/gm0s1d      /var            ufs     rw              2       2
/dev/acd0               /cdrom          cd9660  ro,noauto       0       0

I then rebooted the system. However, it booted from ad4 instead of ad6. That's a problem. It needs to boot from ad6. I was able to fix this in the BIOS. But not before playing around and trying to find a solution from the boot loader prompt. I found this would work:

boot disk2s1a:/boot/kernel

After getting the system booted from the correct HDD, things went much smoother.

So far, we have set up the mirror and specified the source for the mirror. Now we want to add the other half of the mirror into the system. That other drive is /dev/ad4. I used this command:

gmirror insert gm0 /dev/ad4

After adding the new disk to the mirror, here is the list of the gmirror devices:

# gmirror list
Geom name: gm0
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 4226236689
Providers:
1. Name: mirror/gm0
   Mediasize: 500107861504 (466G)
   Sectorsize: 512
   Mode: r6w5e6
Consumers:
1. Name: ad6
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: DIRTY
   GenID: 0
   SyncID: 1
   ID: 533080279
2. Name: ad4
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r1w1e1
   State: SYNCHRONIZING
   Priority: 0
   Flags: DIRTY, SYNCHRONIZING
   GenID: 0
   SyncID: 1
   Synchronized: 0%
   ID: 438208982

#

Yes, ad6 is dirty. That's to be expected. At any given point, there is probably some data to be written to disk. That's all that means. Wait a while. Try again. Perhaps it won't be dirty.

You will see that ad4 is synchronizing. That took a few hours. These are 500GB HDD.

The status command gives a more succinct output

# gmirror status
      Name    Status  Components
mirror/gm0  DEGRADED  ad6
                      ad4 (6%)

Once everything is synced up, and the initial mirroring process has completed, this is what I saw:

# gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ad4
                      ad6

gmirror is cheap, simple, and reliable. I'm using it on a number of my machines at home. It is relatively easy to set up and, more importantly, is stable.

Enjoy.