I recently setup a new server at home to act as a NFS share for backups of other systems, shared files, etc. I had initially planned to use mdadm for a RAID, but after some research decided that btrfs would provide better flexibility and protection against bitrot. Some of the stored data is archived digital photos from years ago, which will be kept for years into the future. Protecting these from bitrot, drive failures, or system failures is critically important to me. Btrfs can do RAID1 by itself and since it does checksums for each block, can detect bitrot and fix it. I opted to use Fedora for this system instead of CentOS7 largely due to recent kernels having better btrfs support.
After a fresh Fedora Server install using full disk encryption with LUKS, I erased two 1GB disks and created a single partition on each,
/dev/sdc1 which will be made into a LUKS encrypted btrfs RAID1.
I wanted the full system to be encrypted with one passphrase at startup unlocking all drives. This can be done by using key files for the non-system drives. (Note that these will be stored on the root partition and accessible in clear text anytime the system is running.) First, create the directory where the key files will be stored, and then create the key files themselves with some random data - this could be done a few ways but I chose to pull 500 characters of base64 from /dev/urandom.
mkdir /etc/luks-keys cat /dev/urandom | base64 | head -c 500 > /etc/luks-keys/sdb1key cat /dev/urandom | base64 | head -c 500 > /etc/luks-keys/sdc1key
Next, use cryptsetup to create the LUKS volumes on each drive, using those key files, typing all caps YES after each command:
cryptsetup --key-file=/etc/luks-keys/sdb1key luksFormat /dev/sdb1 cryptsetup --key-file=/etc/luks-keys/sdc1key luksFormat /dev/sdc1
In order to decrypt and map these LUKS devices at boot, add lines to
/etc/crypttab containing the desired device name, UUID, keyfile path, and in this case
luks, to force LUKS mode. If
/etc/crypttab doesn’t exist, create it. Obtains the UUIDs for each partition with:
cryptsetup luksUUID /dev/sdb1
Then add lines to
data1 UUID=35fd668d-9e1a-4b61-8ac9-8883b21c3c23 /etc/luks-keys/sdb1key luks data2 UUID=b8408310-cb8c-47c0-a0a5-f3d0a28d6fd5 /etc/luks-keys/sdc1key luks
At this point the LUKS volumes should automatically map on startup. You could map them manually with
cryptsetup --key-file=/etc/luks-keys/sdc1key luksOpen /dev/sdc1 data1
but I prefer to test that things are working correctly up to this point by rebooting. Upon startup the device names specified in
/etc/crypttab should be listed in the output of
ls /dev/mapper along with any other encrypted block devices in use.
Next, creating the btrfs-RAID1 filesystem across those two devices. These options tell btrfs to use RAID1 for both data and metadata, not specifiying them would default to RAID0 for data, and RAID1 for metadata.
mkfs.btrfs -m raid1 -d raid1 /dev/mapper/data1 /dev/mapper/data2
Mounting the new filesystem manually (only one mapped device needs to be mounted):
mkdir /mnt/data mount /dev/mapper/data1 /mnt/data
btrfs filesystem show should now display:
Total devices 2 FS bytes used 640.00KiB devid 1 size 931.51GiB used 2.03GiB path /dev/mapper/data1 devid 2 size 931.51GiB used 2.03GiB path /dev/mapper/data2
btrfs fi df /mnt/data should now show:
Data, RAID1: total=1.00GiB, used=512.00KiB System, RAID1: total=32.00MiB, used=16.00KiB Metadata, RAID1: total=1.00GiB, used=112.00KiB GlobalReserve, single: total=16.00MiB, used=0.00B
In order to automatically mount at startup, a line needs to be added to
/dev/mapper/data1 /mnt/store btrfs defaults,device=/dev/mapper/data1,device=/dev/mapper/data2 0 0
Reboot to test. Note: If your fstab file isn’t right and your system won’t boot, you can press
e at the GRUB screen, locate the line starting with kernel or linuxefi and append
init=/bin/bash to the end, then hit
ctrl-x to boot. This will get you a shell prompt where you should be able to go modify your broken fstab file.