Providing practical examples since 1998

Things look quiet here. But I've been doing a lot of blogging at
dan.langille.org because I prefer WordPress now.
Not all my posts there are FreeBSD related.
I am in the midst of migrating The FreeBSD Diary over to WordPress
(and you can read about that here).
Once the migration is completed, I'll move the FreeBSD posts into the
new FreeBSD Diary website.

Jails are a useful tool for achieving a certain level of virtualization.
I have used jails in the past for creating sandboxes for testing of
the Bacula project. Today, I will
be setting up 7 jails, each for regression tests on different configurations.
In the process, I'll show you some shortcuts that Ryan Lortie (desrt) told
me about. It saves time. I'll also share some scripts that will help with
the setup of various jail items.

This article should be read in conjunction with
my other jail articles. It won't stand
alone unless you are already familiar with jails and how to configure them.

"make world" consists of two main steps: make buildworld && make installworld.
There is no advantage in doing the buildworld part again. That has already
be done.

During the building of my first jail, I tried this:

export D=mysql41.example.org

This fail, because the install could not find the directory mysql41.example.org.
I should have included the full path to the directory: /usr/jail/mysql41.example.org.
To complete the installation of the jail, I issued this series of commands:

There are a few things that I like to configure within the jail before
starting it. The files in the jail environment are accessible from
the host environment. So it is a simple matter of using your
favourite editor to make the changes. Here are the files I altered
in /usr/jail/mysql41.example.org:

etc/rc.conf - added: sshd_enable="YES"

etc/resolv.conf - added entries to get DNS working

etc/syslog.conf - changed the "*.err;kern.warning;auth.notice;mail.crit"
line to echo to /var/log/messages instead of the console

It would be easy to script that. For example, here is the script
I used to do the installworld for each jail:

Similar scripts can be created for populating the ports tree. You
may think that a single shared ports tree might be sufficient for multiple
jails. I've decided to go with a distinct tree for each jail. I'll be
building different ports concurrently in each jail. I don't want any
interference caused by another jail.

Setting up the host environment

Setting up multiple jails involves adding several entries to /etc/rc.conf,
some of which specify defaults for all jails. The default settings in my
server are:

It can be tedious to setup thse values. So a script can be useful
and it could be based upon the script that appears above.

PostgreSQL in a jail

Running PostgreSQL in a jail is interesting. There are shared memory issues which can
be resolved by setting security.jail.sysvipc_allowed via sysctl. I tried that.
I failed. The problem I encountered during initdb is:

I decided to double the settings in /etc/sysctl.conf on the host system:

kern.ipc.semmni=512
kern.ipc.semmns=1024
kern.ipc.semmnu=512

But that did not solve the problem. Stopping PostgreSQL on the other two jails did.
This allowed the initdb to complete. But I was still unable to get all the
PostgreSQL servers running in each jail.

After a few more trial and error attempts, I discovered that the settings I had in
/boot/loader.conf contained syntax errors. Instead of SETTING=VALUE, I had
SETTING: VALUE. Fixing that, and rebooting, allowed me to get all five instances
of PostgreSQL running concurrently. But there was a bit more to it than just that.

I tried various things. I tried running each postmaster on a different port
(didn't help). I also tried running each postmaster as a different GID (that did
help). That is the key. I will explain.

Shared memory and UID

PostgreSQL makes use of shared memory. When running multiple instances of PostgreSQL
the shared memory for one instance can be stomped on by another instance. That's not
nice. The key to avoiding this is using a different UID for each instance. You can see
that here:

I used a UID that would relate to the version of PostgreSQL that was running.
For example, UID=1073 is PostgreSQL version 7.3. There is no need to follow
this convention. I did it merely because I could.

When I changed the UID in each jail, I first made sure PostgreSQL was not running
in that jail. Then I ran vipw and altered the UID and exited. Then I did a chown
to ensure the file attributes were owned by the new UID:

chown -R pgsql /usr/local/pgsql

Then I restarted PostgreSQL. No more memory stomping. The symptoms
of this memory stomping (a term I made up and probably does not reflect
reality) are:

I am confident that I am over allocating resources here. I kept increasing the
values and trying them, without success. Then I discovered my syntax error.
I didn't want to reverse-engineer my changes so I have left them as they appear
above. You may wish to use lesser values.

You may be interested in the host system dmesg output. As you can see, this
isn't a very high spec system. It has only 512M of RAM.

With a bit of thought, you can figure this out. top was running in the host
environment. The UID for pgsql has been changed in the jail. The host environment
has no knowledge of UIDs from the jails. When top cannot convert a UID to a username,
it displays the UID. This is fine. An added benefit is seeing exactly which jail
is still running. In this case, it's PostgreSQL 8.0, 7.4, 7.3, and 8.1.

Jails - just right for this

Jails are just right what for what I'm trying here. I certainly do not want
8 servers sitting around the house, just for regression testing of Bacula.
I'll be writing up some scripts to automate much of this testing and to upload
the results to a website.