LUKS btrfs RAID 1 Setup

I recently setup a new server at home to act as a NFS share for backups of other systems, shared files, etc. I had initially planned to use mdadm for a RAID, but after some research decided that btrfs would provide better flexibility and protection against bitrot. Some of the stored data is archived digital photos from years ago, which will be kept for years into the future. Protecting these from bitrot, drive failures, or system failures is critically important to me. Btrfs can do RAID1 by itself and since it does checksums for each block, can detect bitrot and fix it. I opted to use Fedora for this system instead of CentOS7 largely due to recent kernels having better btrfs support.

After a fresh Fedora Server install using full disk encryption with LUKS, I erased two 1GB disks and created a single partition on each, /dev/sdb1 and /dev/sdc1 which will be made into a LUKS encrypted btrfs RAID1.

I wanted the full system to be encrypted with one passphrase at startup unlocking all drives. This can be done by using key files for the non-system drives. (Note that these will be stored on the root partition and accessible in clear text anytime the system is running.) First, create the directory where the key files will be stored, and then create the key files themselves with some random data - this could be done a few ways but I chose to pull 500 characters of base64 from /dev/urandom.

mkdir /etc/luks-keys
cat /dev/urandom | base64 | head -c 500 > /etc/luks-keys/sdb1key
cat /dev/urandom | base64 | head -c 500 > /etc/luks-keys/sdc1key

Next, use cryptsetup to create the LUKS volumes on each drive, using those key files, typing all caps YES after each command:

cryptsetup --key-file=/etc/luks-keys/sdb1key luksFormat /dev/sdb1
cryptsetup --key-file=/etc/luks-keys/sdc1key luksFormat /dev/sdc1

In order to decrypt and map these LUKS devices at boot, add lines to /etc/crypttab containing the desired device name, UUID, keyfile path, and in this case luks, to force LUKS mode. If /etc/crypttab doesn’t exist, create it. Obtains the UUIDs for each partition with:

cryptsetup luksUUID /dev/sdb1

Then add lines to /etc/crypttab.

data1 UUID=35fd668d-9e1a-4b61-8ac9-8883b21c3c23 /etc/luks-keys/sdb1key luks
data2 UUID=b8408310-cb8c-47c0-a0a5-f3d0a28d6fd5 /etc/luks-keys/sdc1key luks

At this point the LUKS volumes should automatically map on startup. You could map them manually with

cryptsetup --key-file=/etc/luks-keys/sdc1key luksOpen /dev/sdc1 data1

but I prefer to test that things are working correctly up to this point by rebooting. Upon startup the device names specified in /etc/crypttab should be listed in the output of ls /dev/mapper along with any other encrypted block devices in use.

Next, creating the btrfs-RAID1 filesystem across those two devices. These options tell btrfs to use RAID1 for both data and metadata, not specifiying them would default to RAID0 for data, and RAID1 for metadata.

mkfs.btrfs -m raid1 -d raid1 /dev/mapper/data1 /dev/mapper/data2

Mounting the new filesystem manually (only one mapped device needs to be mounted):

mkdir /mnt/data
mount /dev/mapper/data1 /mnt/data

btrfs filesystem show should now display:

Total devices 2 FS bytes used 640.00KiB
devid    1 size 931.51GiB used 2.03GiB path /dev/mapper/data1
devid    2 size 931.51GiB used 2.03GiB path /dev/mapper/data2

And btrfs fi df /mnt/data should now show:

Data, RAID1: total=1.00GiB, used=512.00KiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=112.00KiB
GlobalReserve, single: total=16.00MiB, used=0.00B

In order to automatically mount at startup, a line needs to be added to /etc/fstab

/dev/mapper/data1	/mnt/store	btrfs	defaults,device=/dev/mapper/data1,device=/dev/mapper/data2 0 0

Reboot to test. Note: If your fstab file isn’t right and your system won’t boot, you can press e at the GRUB screen, locate the line starting with kernel or linuxefi and append init=/bin/bash to the end, then hit ctrl-x to boot. This will get you a shell prompt where you should be able to go modify your broken fstab file.

References