IOCaging up the RabbitMQ

Here, I’ll show you how to set up rabbitmq, a well-known open source
message queue system, based on Erlang/OTP, in a container, using a tool
called iocage that runs on FreeBSD, using zfs and jails that have been
in base FreeBSD for a decade or longer, connected via tunnels between
endpoints using spiped, a robust and reliable tool designed specifically
for reliably & securely tunnelling network services across unsecured
networks.

I’ve updated this for FreeBSD 10.3-RELEASE and iocage 1.7.4.

Background

I host my servers on real hardware, mainly because the price and performance
is just so much better than virtualised ephemeral cloud services — I’m
guaranteed my ECC RAM, and have no concerns about competing with some other
transient cloud-induced heisenbug not of my own making. As I’m stingy, I
refuse to pay for an additional IPv4 address space, and I want to re-use the
IPv6 allocation that comes with my physical server.

In particular, staying secure, and using only IPv6, introduces some
twists and complications which make this post worth writing.

Creating the iocage

iocage is simply a set of wrapper scripts around FreeBSD jails, a secure
and trusted way of creating lightweight virtual machines that share the same
kernel as the host operating system. This makes them very fast, and we rely
on the tried and tested jail functionality to keep things secure. Jails were
first introduced at the SANE conference in 2000. Note the date – this is 15
years old tech by now.

The jailed filesystem is actually stored as a ZFS dataset, which provides
high-speed, compressed block IO, and can be snapshotted, or transferred
between servers as backup, or to bootstrap remote nodes. ZFS is a stable and
mature filesystem that was ported from Illumos (ex OpenSolaris) into
FreeBSD, starting in 2007. Note the date again. This is solid boring
mainstream technology.

As a bonus, iocage stores all attributes related to the jail as properties
on the ZFS dataset itself, so what we end up with is a fully writable, easily
cloned and transferred, bootable virtual machine. Very much like docker
images, except with less excitement, with significantly better performance,
including direct access to FreeBSD’s network stack, using a battle-hardened
filesystem, and security that has seen over a decade of solid production
use.

There’s no real reason why you have to use the latest FreeBSD release, as
iocage will work on 9.x without issues, but if you are interested later on
in trying out the sysutils/docker-freebsd port, then 10.2 includes the
required latest 64-bit linux emulation, which allows you to run docker
images directly on FreeBSD, using the clever freebsd-docker project. But
that story is for another day.

Bring out the Cage

Before we can get into iocage, we should have a few things set up.
I’m running:

FreeBSD 10.3 amd64 with dual IPv4/IPv6 stack installed to ZFS zpool

unbound caching DNS server accessible on ::1/8 and 127.0.0.0/8

sudo access as there is much root usage going on here

Let’s install iocage, using the development package, start the service,
which automatically creates the zfs dataset (to contain relevant FreeBSD
release images, iocage templates and clones, as well as your jails), and grab
the latest release from a fast mirror near me for future jail creation.

Compare iocage list and what you can see under /iocage/jails/. Again
these are just standard ZFS datasets, you can alter, tweak, and edit just
as usual.

You can also take a look around from inside the as-yet not running jail via
iocage chroot rabbit /bin/sh it’s just like a standard FreeBSD system.

Gild the iocage

No cage would be complete without some fancy trimmings to make our processes
enjoy their confinement. Packages, config files, and a few users are all
that’s required.

Add the packages

The packages are pretty standard, except I use custom builds using an amazing
tool called poudriere which knocks the socks off debian and rpm based
packages. Another story for another, another, day.

FreeBSD 10’s new pkg tool allows me to install from the host system
directly into the jailed filesystem. You can look around from the host
using iocage to look up the uuid that the filesystem’s name is based
upon:

pkg 1.7.2 correctly creates the users inside the jail, which didn’t work
correctly in the previous version of this post, where I needed to create these
by hand.

Again note the base utilities pkg and pw all support chroot work out of
the box. This is one of the secret sauces of the BSD derived operating
systems - userland and kernel are developed and shipped together.

Add RabbitMQ config files

Mostly these are nothing special, however there are a few tricks here to
confine Erlang/OTP’s distributed name service daemon epmd to only loopback
addresses within the jail, and to use the IPv6 address for rabbitmq’s
user-facing functionality

Feel free to use a smaller set of plugins, or a less restrictive set of
permissions. The intent here is that the configuration of the iocage, and
the decreased permissions of the jail user, prevents an attacker from
changing the rabbitmq configuration. An even stronger configuration would
be to put the config files on a read-only ZFS partition. This would require
an attacker to break out of the rabbitmq user to the jail root, & out of the
jail into the host OS. Let’s hope my family photos are just not that
interesting.

While it’s possible to set up Erlang and RabbitMQ for SSL, my experience
with SSL support in OTP has been unreliable and variable between releases.
Also, keeping a CA setup is problematic in itself, because x509 certificates
are frankly a confusing pile of steaming crap. A simpler alternative is
presented, using a single symmetrics key providing end-to-end encrypted
tunnels.

You can read more about spiped elsewhere, but its provenance is from a
highly respected cryptographer with significant practical experience, and
its small codebase significantly reduces the chance of both bugs, and of
exploitable design and implementation errors.

In comparison to autossh or similar tunnel tools, spiped has only 1
function: securing connections. It supports a star model (like a webserver),
decrypting individual connections from different remote peers, handles TCP
heartbeats and timeouts per peer, and maps inbound or outbound ports to
local ones.

Installation is embarassingly simple, and there is just 1 file to configure,
along with generation of the symmetric key and transferring that to each
end.

It is possible to use different keys for the rabbitmq admin interface on
port 15672, instead of the normal user interface on port 5672, or to create
multiple pipes for different servers to connect with. Exercise to reader and
so forth.

Open the Warrens

Before we do that, let’s recap:

rabbitmq is running as a normal unprivileged user inside a FreeBSD jail

rabbitmq is available on IPv6 loopback only for users & management

external access to rabbitmq is on the usual ports, but only via spiped

These processes will just disappear into the background, but could be run by
some master daemon process if preferred.

Show me the GUI

In your browser, visit http://localhost:15672/#/nodes/rabbit%40localhost to
see your rabbitmq admin remote console from inside the jail delivered over a
secured piping hot connection!

Thoughts

Clearly this isn’t an apples-to-apples comparison with docker, but iocage and
the FreeBSD systems we touched are well-documented, very stable, and high
performance. While I’ve used the latest release of FreeBSD all of the above
works just fine on FreeBSD 9.x, and probably even 8 if you have to. We are
not pushed into running a custom kernel to gain performance nor features.

Finally, iocage itself is straightforward shell scripts, so modifications or
understanding is easily gained, and there are no 3rd party registries or
services that you have to rely on. Feel free to use an HTTP server of your
choice, or host your images on some fancy S3-like cloud service.