Diskless, Low-Form-Factor OpenBSD Systems

In a previous article we built a tiny OpenBSD system out of a Soekris miniature PC, a bootstrap workstation, and a Compact Flash (CF) card. While this combination works nicely for many purposes, once you have Soekrii scattered all around your network, managing the CF cards
can be annoying. Replacing the CF cards with a diskless boot system eases
management problems. One modest server can manage many small diskless devices,
and it's possible to do much of the system administration on the server instead of on
the devices. This time, we'll build an OpenBSD diskless device; netbooting FreeBSD and NetBSD will have their own future articles.

Your server must be a NFS server capable of running dhcpd(8),
rarpd(8), and tftpd(8). The server operating system
is irrelevant; any BSD, Linux, or even a commercial UNIX will work. I chose to
use a FreeBSD server, simply because I had spare capacity on a server-grade
system. (My OpenBSD bootstrap station is a Celeron 433 with a 10GB hard drive. The original desktop user complained that it was unusable, but the problems disappeared when we removed the original operating system. It's perfectly adequate for any free UNIX, but not exactly what you want in a server.)

Experimenting with an NFS server, rarpd, and tftpd
probably won't affect your environment, but doing unfamiliar things to the DHCP
server most certainly can, so the main corporate LAN might not be a wise choice
for your test bed. For my initial experiments in diskless booting, I installed a
second network card in my diskless server and ran a crossover cable between
that NIC and the Soekris box. Running a private DHCP server on that interface
allowed me to experiment freely without corrupting the corporate network.
While developers asking, "Why did my machine boot something other than Windows?" would be amusing, it would cause too many meetings.

Soekris Preparation

To configure your Soekris box to run properly diskless, you need the MAC
address of its Ethernet port. Find this by booting the Soekris with a CF
card and running ifconfig(8), or by using a packet sniffer such as
tcpdump or Ethereal. When the Soekris boots, it will attempt to grab its IP
address via DHCP; you can always watch the DHCP logs.

All of the rest of our work takes place on the server. (That's kind of the point, you know.)

Bootstrapping

In the first stage of a diskless boot (as described in diskless(8)),
a PROM or stage-1 bootstrap fetches a boot program from the diskless boot
server. Where can you find such a bootstrap for OpenBSD/i386? Google reveals
a variety of sources, but I chose to go with GRUB for OpenBSD. Cedric Berger has patched GRUB modified to disklessly boot OpenBSD. (This site also has GRUB versions modified to netboot OpenBSD on an Intel "fxp" network card, if you're interested in playing with diskless operations on a standard PC.) Grab the
pxegrub.sis-tty-19200 file and put it somewhere safe.

The Soekris will use both dhcpd(8) and rarpd(8) to
do its initial configuration. rarpd will provide the basic IP
address information, while dhcpd will provide boot-time
configuration information. Let's start by configuring rarpd.

rarpd(8) handles reverse ARP requests. In a normal ARP
request, a machine has an IP address and requests the matching Ethernet MAC
address. In reverse ARP, a machine has a MAC address and requests the
corresponding IP address. The Soekris knows its own Ethernet MAC address and
wants an IP address.

Different rarpd(8) implementations have different features and
requirements; cursory checks for FreeBSD, OpenBSD, and Red Hat Linux show
different behaviors and different command-line flags. For example, the FreeBSD
rarpd I used expected to be able to provide a boot loader to the
client, and by default ignored requests for which it had no boot loader. I
don't want to use rarpd(8) to provide the loader information, so I
had to use the -s command-line switch to tell the program to
provide answers only for those MAC addresses for which it had a mapping. Red Hat
Linux also defaults to checking for a bootable image, but uses a different
command-line switch to just provide service. OpenBSD's rarpd(8)
simply provides the RARP mapping service without trying to provide a bootable
image.

I won't discuss differences in every program we require under every
operating system; the point is, check your server's man pages!

All common rarpd(8) implementations use /etc/ethers to map MAC addresses to hostnames. My /etc/ethers contains the single line:

00:00:24:c1:35:18 soekris-diskless.blackhelicopters.org

The hostname soekris-diskless.blackhelicopters.org must be
available in either DNS or /etc/hosts, so that the RARP server can get
the information.

DHCP Configuration

Next, the diskless machine will attempt to fetch its boot loader information
from the DHCP server. Here, I'm using isc-dhcpd3.

While we don't need to group all of our diskless clients together, it will make further expansion simpler. The filename keyword means "you should go grab this file from the TFTP server." All of our Soekrii need to know about it, so it goes under the group heading.

Each diskless client also needs a separate entry for its MAC address and its
IP address.

TFTP Configuration

Once the diskless Soekris has figured out its IP address and where to get
its boot loader, it will attempt to grab that boot loader via TFTP. All modern, free, UNIX-like operating systems include a TFTP implementation, usually run out of inetd(8). Again, the manual pages for all of the tftpd(8) servers I checked have slightly different options. On my
FreeBSD server, I decided to use /var/tftpboot as the TFTP root
directory, symlinked /tftpboot to it, and then set up
/etc/inetd.conf like this:

tftp dgram udp wait root /usr/libexec/tftpd tftpd -lns /var/tftpboot

The -l flag tells tftpd(8) to log all requests,
which is very helpful when debugging. The -schroots tftpd(8), and the -n tells tftpd to be as quiet as possible.

Now, copy pxegrub.sis-tty-19200 into the tftpboot
directory. The DHCP server will tell the Soekris to grab and run this file.
pxegrub needs a configuration file: grub.conf.
Here's a working example.

While you shouldn't have to change any of this for a Soekris, you might wish
to edit the kernel line. This example makes pxegrub
look for a file called bsd on the tftpd server and
attempt to load it as the kernel. Switch between kernels by changing this
entry.

If you're not using a Soekris and wish to change some of the settings (for
example, to use the monitor and keyboard instead of a serial console), check
the GRUB configuration manual for guidance. Note that the default GRUB configuration file is called menu.lst, not grub.conf;
while pxegrub has changed the configuration file name, the
settings within the file are all identical to standard GRUB.

Final Setup

The kernel will boot the system and should recognize its basic hardware. At
the point that it needs to mount a root file system, the system will automatically
broadcast a request for its boot parameters across the local network. Your NFS
server needs to respond to this request via bootparamd(8).
bootparamd's entire reason for being is to answer broadcast
configuration requests. These configurations are stored in
/etc/bootparams.

The first entry on a line is the fully qualified host name of the diskless
machine. In this example, our second entry is the path to the NFS-exported
root directory for the diskless machine. You can also export swap and dump
device information, but we only need the root directory to start.

bootparamd(8) requires rpcbind(8). On FreeBSD, I
run both of these daemons with the -s flag, which causes them to
drop privileges as soon as possible. (One mistake I made initially was to have
rpcbind attach only to the IP address of the server machine, which
precludes it from answering broadcast requests. The lesson is, don't be too
secure too quickly when you're trying to make this work!)

Exports? That's right, we need NFS! Fortunately, this is very easy to set
up for a single client. The root of our file system is in
/var/obsd/diskless-soekris/root and we only need to export it to a
single host. Here's an /etc/exports for our FreeBSD server.

/var/obsd/diskless-soekris/root -maproot=root 192.168.1.88

We'll also have to start the NFS server daemons at boot. By the time we're
done, the FreeBSD server has the following set in /etc/rc.conf:

Put the proper interface name in the rarpd_flags field for your
server.

Filesystem Setup

You then need an OpenBSD file system under your exported root. That's
actually quite easy to do. You could just mount the root directory from your
OpenBSD bootstrap station and copy the entire system with tar
-xvpf, but that will give you everything on your bootstrap station.
(The -p flag is important; it preserves ownership information.)
Another way to get a small image is to use the file system prepared for a
Compact Flash system; you'll have only the minimal files required, but can
easily add more. Copy over the /etc/rc.* files from your
bootstrap station anyway, as OpenBSD's startup system has specific features for
diskless operations. You will also have to do some debugging of the startup
process, however, and create some directories under /var if you
choose this option. Finally, you could simply extract some OpenBSD
distribution tarballs in that directory and install vital /etc
files from your bootstrap station.

The DISKLESS Kernel

OpenBSD provides a kernel configuration specifically for i386 diskless
operations. It's called DISKLESS. Remember, the OpenBSD folks are primarily
interested in supporting the GENERIC kernel; you'll have minimal support if you
use a non-GENERIC kernel. However, in my experience, the DISKLESS kernel configuration
is quite stable and reliable.

As long as you're on a custom kernel anyway, you can make your life a little
easier by equalizing the serial console speeds. The Soekris uses a serial
console speed of 19200, while OpenBSD defaults to 9600. If you connect to the
serial console at 19200, the Soekris boot messages will be legible, but the
OpenBSD console will show only garbage. If you connect at 9600, you'll get the
OpenBSD messages, but the Soekris information will be illegible. If you add a
few kernel options to make OpenBSD speak at 19200, everyone will just magically
get along.

option COMCONSOLE
option CONSPEED=19200
option CONSOLE=com0

With this addition, our Soekris box is trivial to manage. When the time
comes to perform a system upgrade, you can move aside the current root
file system and replace it with the new one. This also makes reverting a bad
upgrade trivial. The Soekris can mount the root file system as read-only, and you
can still easily edit files on the server. Soekris boxes make excellent
firewalls or other types of network devices; with diskless operations, they can
also become small network servers, with very little increase in administrative
overhead.