FreeBSD jails with ezjail

At work, we've decided to virtualize all of our FreeBSD production systems with jails. We're hoping this will help with upgrades and potential system failure, as we'll be to move virtual systems around, where neeeded, with relative ease.

While setting up jails with FreeBSD directly isn't a difficult task, there is one major limitation, if you're going to have multiple jails on the same host. The entire base system (/bin, /usr/bin, etc), needs to reside within each jail. Using a very well written set of shell scripts to help manage our jails, however, we're able to mount, with the help of nullfs in FreeBSD 6+, read-only, our entire base system.

Here, I'll describe in great detail the entire process I used to install ezjail, build our basejail, and create a flavour [sic] that meets our needs.

With this setup, we have a base system with postfix as our MTA. Skip the appropriate parts if you use sendmail or another MTA.

Build host system

Install/Update FreeBSD

To begin, we build out host FreeBSD system. This includes performing a full installation of FreeBSD 6.2, and doing all the updates. Whereas freebsd-update is a rather nice binary update tool, it is advisable to build/install directly from source, and you'll end up doing it for your jails anyways. You won't be saving yourself any time doing binary updates.

Install Common Ports

Once we have a basic install of FreeBSD, our network requires we install postfix with sasl to send email. This allows email to be sent to a user with a simpel 'mail <user>' and everything will get translated correctly. Along with setting up mail, continue with installing any ports that you'll need/want on all your jails, onto the base system. Individual jail ports, such as apache, mysql, etc, should not be installed at this time. Only ports you want on all jails should be installed.

We've installed the following ports:

sudo (security/sudo)

vim-lite (editors/vim-lite)

bash (shells/bash)

rsync (net/rsync)

postfix (mail/postfix)

Jail-Friendly Host Setup

In order to allow for things like ping, etc, we need to change that security.jail.allow_raw_sockets to 1.

echo 'security.jail.allow_raw_sockets=1' >> /etc/sysctl.conf

Install ezjail

Now install the ezjail scripts from the FreeBSD ports tree at sysutils/ezjail. You will be left with an ezjail.sh script, an ezjail-admin.sh script, and an rc script in /usr/local/etc/rc.d. To enable our jails to start at system boot, we needed to add ezjail_enable="YES" to /etc/rc.conf:

echo 'ezjail_enable="YES"' >> /etc/rc.conf

Configure

There is a configuration file for ezjail in /usr/local/etc/ezjail.conf.sample, we want to move that to /usr/local/etc/ezjail.conf and edit the file. Make appropriate changes for your network. We run our own CVS mirror, so our file looks like this:

ezjail_jaildir=/usr/jails
# Location of the tiny skeleton jail template
ezjail_jailtemplate=${ezjail_jaildir}/newjail
# Location of the huge base jail
ezjail_jailbase=${ezjail_jaildir}/basejail
# Location of your copy of FreeBSD's source tree
ezjail_sourcetree=/usr/src
# In case you want to provide a copy of ports tree in base jail, set this to
# a cvsroot near you
ezjail_portscvsroot=:pserver:anoncvs@<our_local_CVS_mirror>:/home/ncvs
# This is where the install sub command defaults to fetch its packages from
ezjail_ftphost=<our_local_FTP_mirror>
# base jail will provide a soft link from /usr/bin/perl to /usr/local/bin/perl
# to accomodate all scripts using '#!/usr/bin/perl'...
ezjail_uglyperlhack="YES"

Change <our_local_FTP/CVS_mirror> for your correct values.

Build basejail

At this time, we should be ready to build our basejail!

Run the following command to build basejail within your configured jail home:

ezjail-admin update -ip

The -i option above tells ezjail that we've already built-world (when we updated FreeBSD on the host system), so it simply does a make installworld to your jail home. Omitting the -i causes this process to take a considerable amount of time because 'world' is built.

The -p option above tells ezjail that we want the ports tree included in our jail. This will take additional disk space equal to that of the size of your ports tree.

When this process is complete, you should have a directory structure similar to this in your jail home (/usr/jails by default):

Building Our ezjail Flavour

Version 1.2 of ezjail introduces the concept of Flavours. A flavour corresponds to a preconfigured Jail. Basically it is a directory being copied recursively over a new Jails root and a script being run on its first startup to do some initialization.

Assuming your jail home is /usr/jails (which we will assume for the remainder), you should have a /usr/jails/flavours/default directory. We want to copy that directory to a name for our own customized flavour. Ours will be called clx.

# cd /usr/jails/flavours
# cp -Rp ./default ./clx

Copy /usr/local

When your jail is finally built, anything within the corresponding flavour directory will be copied over the basejail that is installed. Our first order of business is to copy our entire host's /usr/local directory into our flavour directory. This will get us all those common ports we installed earlier:

cp -Rp /usr/local/* ./clx/usr/local/

Note, this process will probably take quite a while, depending on which ports you have installed.

ezjail.flavour

Inside your flavour's root directory, you should see an ezjail.flavour file. This file is essentially a script that get's run the first time a jail that was created with the flavour is started. In our case, we are going to add some users (and their groups), add the postfix/cyrus users and groups, and perform a few other initial maintenance tasks. Here's our file as an example:

Any other shell commands you need to run to setup your jail can be put in this file.

Building a Jail

Now that we've got all the configuration and defaults set up for our jails, building an actual jail is pretty darn easy. Let's say we want to create a jail using our clx flavour for www.example.com with an IP of 10.0.0.5. Simply use the following command:

# ezjail-admin create -f clx www.example.com 10.0.0.5

Yeah, it's that easy. Once the build is done, it'll take a couple of minutes, we need to assign that new IP to our jail host, and put that IP in /etc/rc.conf to make it persistent through reboots:

# ipconfig <interface> alias 10.0.0.5/16

rc.conf:

ifconfig_interface_aliasX="inet 10.0.0.5/16"

Note: Replace interface with your actual interface and aliasX where X is the incremental IP alias.