Introduction

Understanding the architecture

A diskless boot system is comprised of a couple different component that enables the entire boot process to work.

First the machine requires a boot mechanism. This can be PXE, which is a feature of the network card to boot entirely from the network. This could also be a boot floppy, CD, USB key, or even a small hard drive used only to kickstart the boot process.

If the machine is booting using PXE, the PXE boot code acquires a DHCP/BOOTP address from a DHCP server that contains information on where to find the bootloader. This is similar to a Grub or LILO boot screen normally found on a Linux system. This PXE bootloader defines where the init filesystem (Initrd) and the kernel are located and can be downloaded using TFTP

If the machine is booting using a local media method, the bootloader on that media directs the boot environment to the init filesystem and the kernel.

Once the init filesystem and kernel have been downloaded and loaded onto a Ramdisk, the remainder of the filesystem is mounted via NFS from the server.

Warning: If you are thinking about diskless and have an Nvidia card, there is a known bug installing their driver on such a system (see Installation fails on NFS). This should only apply to building the driver on that system - installing from an RPM should be fine.

Creating a test system

I found that the VMware Player can provide an excellent system to test PXE booting and the network boot in general. The VMware Player provides a virtual PC environment that also emulates a PXE network adapter. By using this environment, you can quickly test changes to the DHCP, TFTP, and the kernel itself. Once you can get this virtual machine to boot, it is a simple matter to get a physical machine working.

Use this PXE.VMX file as a starting point for a machine you can test with

Configuring the DHCP server

In order for any network boot to function, a DHCP server must be running on the network and support the custom configuration options needed for the network boot. While this could be on any operating system, these instructions refer to the Linux DHCPD implementation.

PXE boot

To support PXE booting, the following lines must be added to your /etc/dhcpd.conf file and the dhcpd service restarted

The next-server directive indicates the TFTP server that the system will contact. The filename is the init file that will be loaded to provide the bootloader. This example is specific to the Redhat and Fedora Core implementation and may vary depending on your distribution.

Note: This configuration gives the option to use both pxegrub AND pxelinux, whichever one your box picks up first. I show this option to give a wider group of people the ability to boot, assuming they have the appropriate files.

TFTP server

The TFTP server for your distribution is required for PXE booting and needs to be installed.

Gentoo specific note:
If you blindly emerge "tftp" you will emerge hpa-tftpd. I found that I needed to edit the /etc/conf.d/in.tftpd file and add the -l (listen) option to get the tftpd server to start working.

DHCP server

In order to mount the NFS root partition, the DHCP server will also need to provide a root path and IP address to the booted system

option root-path "/diskless/i386/RH9/";

This option directs the DHCP client to the root filesystem. This example is specific to the Redhat and Fedora Core implementation. Please refer to your distribution documentation for details.

Distribution specific instructions

Mac OS X

Redhat and Fedora Core

Redhat and the subsequent Fedora Core releases provide a fairly straightfoward method of creating a diskless boot environment that can support both PXE and Netboot or Etherboot environments.

These releases provide a redhat|system-config-netboot script which does most of the heavy lifting. It creates a PXE boot image as well as kernel and a snapshot folder for each netboot system. The snapshot folder stores all the host specific files such as /dev, some parts of /etc and /var. The main root filesystem is mounted read only to provide sharing between multiple systems. It uses a pivot-root function to overlay the host specific files and folders on top of the read only portions of the filesystem.

TFTP server notes

The TFTP service is part of the xinetd installation. Enable it by editing your /etc/xinetd.d/tftp file

Set the disable option to no. I also added the -v option so that it provides additional verbose logging to /var/log/messages which can help with troubleshooting problems.

Creating the root filesystem

Creating the root filesystem is basically a matter of copying a working installation that you want to make your client boot to. This is mostly likely a hard disk installation on the hardware you will be booting from.

You can replace the * with a hostname, an IP address or range, or domain name (*.myhomedomain.org) to limit access to the host(s) specified if desired.

Creating the boot images

The system-config-netboot package provides the scripts necessary to create a diskless configuration. It requires the installation of the busybox and busybox-anaconda packages. My target and source system was Redhat 9. The redhat-config-netboot script provided with Redhat Enterprise Linux is massively broken. The system-config-netboot script provided with Fedora Core 4 had some additional python dependencies that I didn't care to work out so I chose to use the Fedora Core 3 script which worked fine.

If you already have a boot structure in place, be aware that this script will blindly overwrite your previous pxelinux.cfg/default file. Take a backup of the entire directory BEFORE starting the script.

Running this script will first ask you to create a diskless configuration from an installed kernel. If you built a kernel with the options outlined earlier and installed it on the server then choose that kernel. I found it is easiest to use the same kernel for both the server and the diskless client whenever possible. It makes buildling kernel modules for the diskless kernel alot easier because the root filesystem on the diskless machine is read-only and it can make some installers throw fits, like nvidida.

The linux-install folder under your /tftpboot directory will be populated with the kernel specific files including the initrd and the kernel itself. It will also create the snapshot folders.

Next, click New on the next window to appear to create a new host identifier which populates the snapshot folder with information specific to this diskless host that will be booting from the root partition created earlier.

However system-config-netboot-0.1.41-1.FC6 does not work flawlessly on an FC5 or FC6 system. To make it work, you need a couple of changes.

1) FC5 & FC6: Add "ramdisk_blocksize=1024" to the append line of the boot stanza below.

2) FC5 & FC6: SELinux can cause problems with logging in to the system when it is netbooted. Disable it by adding "selinux=0" to the append line of the boot stanza below.

3) FC6 only: Include /sbin/mount.nfs in the initrd.img after it is generated. This can be done as follows (as root).

4) FC5 (with newer kernels) & FC6: Several people have found that the snapshot directory is not being mounted rw using the newer kernels. The following is a fix involving modifying the disklessrc file which is run after the kernel boots. It references the same initrd.img file above,only do this step IF you are having this problem, otherwise just skip it.

bzImage-2.6.19.diskless is the kernel image to load which must be present in /tftboot

Note: In this scenario no initial ramdisk (initrd) is required but the kernel must have the following options built in (not as modules):

The driver for the booting NIC

CONFIG_IP_PNP

CONFIG_IP_PNP_DHCP

CONFIG_NFS_FS

CONFIG_NFS_V3

CONFIG_NFS_V3_ACL

CONFIG_ROOT_NFS

And the root-path DHCP option specified as something like

option root-path "192.168.0.1:/netboot/frontend";

192.168.0.1 is the IP address of your NFS server.

/netboot/frontend is the export that will become root on the remote server.

Configuring a netboot system

If you are using a CD, USB key, or other media configure your bootloader appropriately. Copy the initrd.img and vmlinuz files from the /tftpboot/linux-install/RH9 folder to your installation media. In my work I happened to use a small local hard disk to jumpstart the process and had my /etc/grub.conf file configured as

The (hd0,0) in the initrd line was very important because once the kernel started, it didn't understand where to find the initrd file unless the specific devices was provided. This is different than the PXE method which knows where both reside.

Gentoo Specific with configuration file examples

DHCP Server

The default /etc/dhcp/dhcpd.conf from emerge dchp proved to be more data than necessary for the simple needs of DHCP I need for *only* the PXE machines on my network (I have a DHCP server on my router that does the DHCP for the entire network, I don't need a second one ;) ). So, I junked the original:

mv /etc/dhcp/dhcpd.conf /etc/dhcp/dhcpd.conf.orig

And created this new one, I'll try to break down the sections below the code box:

This example assumes you do not need the DHCP functionality for your network from this DHCP server, it's only DHCP'ing for your PXE boxes, hence the comment on the domain-name option. The subnet and netmask will be particular to your network, this one assumes you have a simple 192.168.1.1 based network (192.168.1.0/24). I gave a range of 1 IP rather than specifying fixed-address in the 'host mythtoo {' section; why? Because I'm eccentric, or maybe it was an issue with obtaining a valid /etc/resolv.conf ;)
The other options should be pretty self-explanatory:
hardware ethernet is the MAC address of the client (the diskless machine)
option option-150 was my life saving option that i uncovered somewhere online, it was necessary for pxegrub to boot.
option routers was where I specified what routers were on my network, after I did this I was able to access the internet from the PXE box, otherwise I was restricted to the internal network (which, for a mythbox might be great).
next-server is the IP of the server where your TFTP server lives. This happens to be the same server I have everything on, but it seems necessary to spell it out anyway.
filename This is where you specify your bootloader, pxelinux.0 for PXELinux obviously, and pxegrub for Grub :D Notice the full path, not a chroot'd path.

TFTP Server

Default emerge of "tftp" pulls in HPA-TFTPD, and is called by in.tftpd. All that was not exactly easy to find if you don't read your emerge output. From there, the default /etc/conf.d/in.tftpd doesn't work, I had to modify it to look like:

# /etc/init.d/in.tftpd

# Path to server files from
INTFTPD_PATH="/mnt/pxe"

# For more options, see tftpd(8)
INTFTPD_OPTS="-s -l ${INTFTPD_PATH}"

Note: The addition of the "-l" option, this enables tftpd to listen (why doesn't it listen by default?). To actually get my PXE client to boot, I had to /etc/init.d/in.tftpd stop which is very odd, but works for me, YMMV.

PXE Bootloading Options

I had a great time figuring out how the PXELinux boot loader works, but once I did, it was solid. By default it will search a chroot'd / directory (the / of your PXE client as specified in your TFTPD file). So, you will be creating a configuration file. You have a slew of options:
You can use the MAC address of the Client, in the form of:
00-01-a1-c1-d1-f1-x1 (note the lowercase)
Or you can use the Hex version of your IP:
C0A8013C (PXELinux comes with an application to convert an IP to Hex: gethostip)
Or any octet short of that (so C0A8013, COA801.. and so on) OR
you can just use the word default.
I chose all 3 ;)
So you create a directory called pxelinux.cfg in your / directory, and inside it you place your configuration files as described above (MAC address, so on), which should contain something that resembles this:

Should be relatively self-explanatory. /boot/bzImage-2.6.16 is the kernel image in the /mnt/pxe/boot directory. nfsroot is the exported directory from my NFS server (my NFS server's IP is 192.168.1.90)

Final points

I also had to emerge nfs-utils to get it to stop complaining about netmount not working. It may take 3-4 tries of booting to get your RC's straightened out to let you through to a login.

Conclusion

Once the system is booted, there may be various tweaks and settings the need to be modified. Working with a diskless system can be tricky but since you have access to the root filesystem on the server and can use chroot /diskless/i386/RH9 to change into the environment it can lighten the load.