This is a continuation of my earlier explorations booting from a LUKS encrypted disk. This time, I'm booting Gentoo Linux from a LUKS encrypted ZFS volume. I won't go into why in detail (others do that at length elsewhere), but I do like the checksumming and snapshotting that ZFS provides. I'm still re-working my backup system to function through ZFS snapshots, but my progress so far is already great.

I've now done this procedure twice for real (my server at home, and the remote server in my Mom's basement to serve as the off-site backup). And of course, I learned and tweaked as I was doing so. I took notes, and I'm doing it a third time in a virtual machine to make sure I got all the details right. The first time I figured out booting from LUKS it was really helpful, even just for me to refer to again later. You'll best start with my Gentoo minimal install LiveCD patched with ZFS support. Any bootable 64-bit ZFS enabled media will do, but that's easy and familiar if you've done Gentoo before. Note however that this is a simple cookbook style reference, not a tutorial. If you don't already know ZFS, you'd do very well to read up on it. I have a collection of links at the end for lots more detail.

Work through the Gentoo Handbook until the Preparing the Disks section. My disk scheme is: three 2TB disks in a RAIDZ1 pool, with LUKS encryption underneath. They're all the same model and thus the same size. Additionally another small disk to hold the (unencrypted) boot. I started testing with a plain old USB flash drive, and later switched to a Compact Flash to IDE adapter as a cheap/small/low power fake SSD. So in my case sda, sdb, and sdc are the main three drives, while sdd is the (USB) boot drive. Setting them up looks like:

(A quick note on the examples before this first one: The shell prompts are colored red, the inputs I type are colored green, and the rest is the output. Your output will likely differ in small details; I'm trusting you to be intelligent enough to figure that out if you're following this as a guide. But I find archiving the output still makes it easier to follow along. Your inputs may differ as well, be careful to make sure you are referencing (e.g.) the proper disk!)

A note here on encryption passphrases. I've elected to use the same passphrase on all three disks. Later, I set it up so that I only have to type this passphrase once upon boot to unlock all three. I suggest you do the same. Also note: yes, I am encrypting the entire raw disk here, on purpose. ZFS does not need partitions, just a block device to store its data. The boot disk is partitioned however, to give GRUB room to install.

Not create a cachefile. These can speed up the zpool import (i.e. "mount") phase. With just three vdevs in the pool, I find this unnecessary, and skipping the complication is better.

-O atime=off

Not record access times.

-O compression=lz4

Turn on compression. This is a good idea. Modern processors tend to have more spare resources than modern disks. This compression algorithm specifically is lightweight, but acheives nice ratios on appropriate content.

Sets the "altroot", i.e. the temporary alternative mount point. This value is appropriate for the Handbook driven install process.

rpool

The name of the pool.

raidz1

The type of the pool.

crypt_sda crypt_sdb crypt_sdc

The devices making up this pool; ZFS can find them just by these short names (which are unlikely to otherwise exist) and their brevity will make other output easier to read.

And in good Unix tradition, produces no output upon success. Now to create the datasets within the pool. ZFS datasets are heirarchical. I've created two top level data sets: root and tmp. The first gets regular snapshots (via zfs-auto-snapshot), which get replicated off site. The latter does not, because it contains only files that are easy to replace (linux kernel source, portage) or are not worth backing up (large scratch files, media archives).

Note that ZFS will auto-mount these data sets at the given mount points (relative to the altroot specified at zpool creation). Fill in the actual name for "user". Also optionally (but recommended) set up swap.

We're doing a standard Gentoo install now, from the installation section. When reaching the configuring the kernel section, use genkernel. We'll need to use a custom linuxrc in order to luksOpen all three of our encrypted drives at the right point (before it tries to mount our ZFS root). We must also build a valid kernel before we can (build and) install the ZFS kernel modules. So:

The result is that the three encrypted disks will be unlocked immediately before the ZFS pool is imported. This technique does unfortunately read in the LUKS passphrase in a shell script, and echo it (via the command line) into cryptsetup. Putting sensitive data like a passphrase onto the command line like this is normally a big security no-no. I'm comfortable with this simply because it exists only inside the initramfs; any attacker capable of monitoring closely enough to catch the passphrase at this point (by its presence on the command line) can mount a simpler and equally effective attack. If you're uncomfortable with that, you can simply omit the reading and echoing of $PASSPHRASE (and thus be forced to type it three times instead).

Some of these flags can instead be permanently set via /etc/genkernel.conf but not (as far as I know) the --zfs nor --linuxrc flags, both of which are critical. The rebuild callback is probably only necessary this first time.

Continue with the handbook. The fstab should be empty besides /boot and swap. Install grub2 into the bootloader, and configure it for booting:

Unfortunately we really do need to dozfs=force. Linux seems unable to cleanly export (i.e. unmount) the root file system partition. Forcing the import is the only way I know to make things work. Continue from handbook section configuring the system. Done!

Appendix: Recovery

Should you ever need to reboot during installation, or later boot from the livecd for recovery, it would go something like this:

The floppy module seems to get auto-loaded even though the machine has no floppy drive, and this causes zpool import to hang while trying to check if it should open a vdev on fd0. So first remove it. The luksOpen lines unlock the encrypted volumes, then we mount all the other required paths and chroot into it.

Use angle braces around a plain URL to auto-link it: <http://www.example.com/>.

Use this format to create a link with different text showing: [An Example](http://www.example.com/).

Use backticks (``), not leading spaces to enclose a code block.

If you are attempting to contact me, ask me a question, etc,
please send me a message through the
contact form rather
than posting a comment here. Thank you.
(If you post a comment anyway when it should be a message
to me, I'll probably just delete your comment. I don't like
clutter.)