ZFS on DigitalOcean Block Storage

DigitalOcean presently has a block storage feature in the beta stage. I’ve been doing some poking around with it using FreeBSD and ZFS, and it makes for an easy to set up and expand storage solution. Here’s a quick walkthrough of how to use block storage volumes with FreeBSD for ZFS, including encryption.

Creating the droplet

First off, we need a FreeBSD droplet. I’d recommend at least 4GB of RAM, as ZFS tends to be very memory intensive, especially if you’re interested in doing block de-duplication. With less than 4GB, FreeBSD will disable ZFS prefetch by default, and under 2GB will likely need special tuning to remain stable. For more on that topic, see the ZFS Tuning Guide in the FreeBSD wiki.

Creating a volume

Now that we’ve got a FreeBSD droplet, let’s create a volume to use with it. We just need to specify the volume size and region, and give it a name. This name is just a label to identify the volume and can be whatever you like.

Just as a quick note, the local SSD for a FreeBSD droplet appears as ‘vtbd0’ and any volumes attached will appear as ‘da’ devices.

Partitioning the volume

While not strictly necessary, since we’ll be using the entire volume for a single filesystem, it’s generally a good idea to put a partition map on the volume. This allows for meaningful labels to be applied among other things. In this case we’re using the GPT format:

% sudo gpart create -s gpt da0
da0 created

Then we create a single partition for ZFS, indicated by the -t type flag. The -l option is just a label for the partition and can be whatever we like. In this case, I’m having it match the volume’s name to help keep things straight:

% sudo gpart add -t freebsd-zfs -l volume-nyc1-01 da0
da0p1 added

Setting up encryption

If you’re not interested in encrypting data on the volume, you can skip this section and just leave the .eli off the device name in later steps.

If you do want to encrypt your data, let’s start by loading the aesni driver to enable hardware accelerated AES encryption:

% sudo kldload aesni

Now we can configure geli encryption on the partition. The -l option here is the key length, which has to be either 128 (default) or 256 for the default AES-XTS algorithm. The passphrase entered here will be needed when the droplet is rebooted to re-attach the encrypted partition. Also note that we’re referring to the partition by the label supplied earlier.

% sudo geli init -l 256 /dev/gpt/volume-nyc1-01
Enter new passphrase:
Reenter new passphrase:
Metadata backup can be found in /var/backups/gpt_volume-nyc1-01.eli and
can be restored with the following command:
# geli restore /var/backups/gpt_volume-nyc1-01.eli /dev/gpt/volume-nyc1-01

Now that geli has been initialized, we need to attach the encrypted partition to the system. This step will need to be done when the system is rebooted. The passphrase entered here has to match the phrase used during initialization.

% sudo geli attach /dev/gpt/volume-nyc1-01
Enter passphrase:

This will set up /dev/gpt/volume-nyc1-01.eli, which is the decrypted version of the partition. Data written to that block device is encrypted and written out to the underlying device.

Setting up ZFS

Now for the easy part, creating the ZFS pool! Since DigitalOcean volumes implement their own data redundancy, there’s no need to create multiple volumes and mirror them, or to run them in a RAID-Z configuration, we can just use the individual volume directly in the pool. We’re using the generic name of ‘tank’ for our pool, but again, it can be pretty much anything you like.

% sudo zpool create tank /dev/gpt/volume-nyc1-01.eli

Since the volume is attached over a network, access to it is going to be slower than the local SSD. In order to minimize the amount of data being written to the device, we can enable compression at the ZFS filesystem layer. This is entirely optional, and can be set on a per-filesystem basis. In this case, we’re using the ‘lz4’ compression algorithm, which is optimized for speed while still giving decent compression. For other options, consult the zfs man page.

% sudo zfs set compression=lz4 tank

At this point, we can now look and see the ZFS pool with it’s total size being slightly smaller than the total volume size, due to partitioning and formatting overhead: