I don't actually have a reason for trying to build a Linux kernel with the CONFIG_PREEMPT_RT patch set. There's no way I can even measure the impact of it. Still, I felt like having a "real-time" Linux box, and set out to make one. Little did I know how difficult it would be to even get started.

A little bit of background. I haven't used Linux all that much, though I've been using various *nix variants for about 20 years. I played with Linux in the pre-1.0 beta days, installed Yellow Dog on my PS3 (before Sony took it away), and played with Ubuntu when it started to get a lot of buzz. In college and at my first job I did a bit in SunOS (yes, I'm so old that I predate Solaris). For much of my career I have used the QNX real-time operating system. I wanted to build a system similar to a QNX box, even though I have no practical use for such a beast. I thought it would be fairly easy. I'm no expert, but I didn't think it would require more than a basic familiarity and the help of Google.

First things first; what do I use as a base system? There are so many distributions that it's difficult to pick one. I've been defaulting to Ubuntu, but I stopped at Maverick, and I wasn't sure I wanted to learn the new Unity interface for this project. I settled on OpenBox as a desktop environment, and that cut things down to ArchBang and CrunchBang. Of the two, CrunchBang seemed like the better route for me since I was used to apt-get and Synaptic. So I installed CrunchBang in a virtual machine using VirtualBox. Installation was no problem. I downloaded the kernel and rt patch. No big. Extracted the kernel and patched it; easy enough. Most packages can be compiled with a three step process:

./configure
make
make install

The Linux kernel, however, is a very large chunk of code, with lots and lots of options. Therefore, it has its own version of "configure." Instead you run "make menuconfig" or "make gconfig", depending on whether you want a console ncurses configuration program or a GTK+ graphical configure. This is where the problems began.

Though I'm perfectly comfortable on the command line, I see no reason not to take advantage of GUI's when they're useful. So I tried "make gconfig." It didn't work. I lacked a dependency. It took me quite a while to find instructions on the Web as to what packages I needed to apt-get. When I did finally discover the correct package names, they failed to install. Very frustrating.

I decided to give up on CrunchBang and try ArchBang. I immediately liked it; it booted quickly, the archbang.org website looked nice, and there seemed to be plenty of instructions out there. I ran into trouble almost immediately. ArchBang creates 4 separate partitions; boot, swap, system, and home. I hate this. Twenty years ago maybe there was a reason to have a separate swap partition; no longer. Same with boot. Separating /home can make a lot of sense, but for a single user development system it's pretty pointless. With the 8GB default hard drive size I had used, I ended up with a home partition of about 500MB, which meant that I couldn't actually extract the kernel in my home directory. Ugh. Fine, whatever, sudo mkdir /work, chown, and chgrp, on with life. Still no good. I had the space, but make gconfig errored out.

Well, at least ArchBang has a nice post-install instruction list that gave me a good idea of what to do next. It mainly involves using pacman (the Arch package manager, similar to apt-get) to update the system. Except it didn't work. I followed the instructions exactly, and the commands failed. I also realized I didn't have the VirtualBox Guest Additions installed. Stymied there as well, because I couldn't figure out how to mount the (virtual) CD-ROM. Fortunately, "pacman -S virtualbox-archlinux-additions" did the trick.

Then I had to go on a business trip. I switched to my laptop. Due to lack of foresight, I failed to move the virtual machine over. I tried to get it over Windows Remote Desktop and failed. I tried to have my desktop upload the (roughly 2GB) file to SkyDrive so I could get it from there. For some reason the file never uploaded. Fine, whatever, it doesn't take that long to install ArchBang, so I started over. I got it mostly up and going, but then at some point I bricked the install. I'm not entirely sure what did it, but probably either "pacman -S linux" or "pacman -S udev" was the culprit. Most advice on the forums seems to be "re-install." I could have tried that, but I didn't see much point. I just started over from scratch, and hoped I could figure out a new set of post-install instructions that would work consistently.

Back at home, I decided maybe I had bitten off more than I could chew trying to get an "expert" distro going. I switched back to Ubuntu. Oneiric doesn't split /home on to a different partition, but that wasn't the problem. After the install has updated everything, I was already out of space thanks to that 8GB default drive size in VirtualBox. 8GB should be plenty of space, but a default Ubuntu install eats almost all of it. Ugh.

Fine, fine, back to ArchBang. I started from scratch again, this time specifying a 20GB virtual disk. ArchBang still split off /home into it's own partition, but now I had about 10GB for it. I more carefully went through the post-install instructions and Google found me work-arounds for problem areas. I got everything in to good shape, but now the VirtualBox guest additions have stopped working. I managed to get the CD-ROM mounted and installed straight from the CD, rather than using pacman to get the ArchLinux specific version. Finally, I had good working system. Clipboard sharing worked, I could mount my VirtualBox shared folders, and my .bashrc was configured well enough to not drive me insane. "make gconfig" worked. Phew!

Notice the "Preemption Model" option. I didn't do a whole lot besides turn on "Fully Preemptible Kernel (RT)." However, I had to go through the config / build process numerous times. I had ignored the advice of starting with a kernel configuration copied from a working system. Literally every site that talks about building the kernel mentions this. Unless you really know what you're doing, start with a known-working configuration. Also, gconfig isn't a very good program. I wasn't sure whether to click or double-click to change an option. (Double-click.) Sometimes it wouldn't let me change an option. I wanted to turn off the I2C drivers, for example, but couldn't. I couldn't change the custom name field from gconfig; I had to go edit the .config file manually.

Eventually I got through a config / build sequence that I felt confident in. The actual build takes a while, about 4 hours on my Core i7 inside a virtual machine. Now, of course, I needed to install the kernel. This is not something I have any experience with. The ArchLinux wiki came to the rescue with a nice set of instructions. Even after this whole process, I don't really know where all this stuff goes, so it's a good thing the instructions worked. Then the scary part; reboot, and hope it works.

Hurray! I am running a custom kernel! Note the "rt" in the "Kernel" line. Twice. Once was automatic, and the one on the end was because I didn't realize it would be there automatically. Also note the slightly older kernel rev; the rt patch only existed up to 3.2.12. When I started, the 3.3 kernel was already out, and while I was working on this 3.2.13, 3.2.14 and 3.3.1 were released. The rt patch for 3.2.13 came out by the time I finished this article, but by that point I was too exhausted to try again.

I'm afraid I started off a bit over-confident. Okay, a lot over-confident, or else I wouldn't have had to write this article. I had started off thinking a trained monkey with access to Google could build the kernel. It turned out to be harder than I thought, but it still isn't rocket science. The main thing to have is a fair amount of patience. And Google.

In addition to all of the help from the ArchBang and ArchLinux documentation and forums, I was also looking over the Linux From Scratch (LFS) book. While there wasn't a specific set of instructions I used from LFS, I did encounter a concept that I should have embraced early on. When trying to a build a system following LFS, it helps to actually follow LFS. The forum abbreviates this to "FBBG." This stands for "Follow Book, Book Good." This was good advice. I know that sounds a bit like saying a trained monkey would have been better off than I was. That's partially true. The point at which to deviate from the path is when (a) the path obviously isn't working, and (b) you know why you're stepping off the path.

So, was it worth it? Probably not. This was more a learning exercise than something useful. It was frustrating and ultimately doesn't have any real benefit. Still, I learned an awful lot, and there was a definite sense of accomplishment. I'll probably continue to run a custom kernel, if only for the whole mountain-climber "because it's there" feeling.

The actual kernel build process:

Download the CONFIG_PREEMPT_RT patch set from https://www.kernel.org/pub/linux/kernel/projects/rt/ Do this before downloading the kernel because we need to know what the latest version of rt is.

Download the matching kernel from https://www.kernel.org/pub/linux/kernel/

Extract the kernel to a working directory.

From within the linux-[version] directory, run "bzcat patch-[version].patch.bz2 | patch -p1

Get a copy of a working .config file. On Arch, this can be done by "zcat /proc/config.gz > /path/to/build_dir/.config"

make O=/path/to/build_dir gconfig

Edit the configuration to your liking. Particularly remember to turn on the Preemption, as it is not on by default.

Configure the boot loader. This is different for each loader, but the short version for GRUB is edit /boot/grub/menu.lst. Copy-paste a working entry and edit it to match your new kernel and the files you put in /boot.

Reboot. Note that there is no evidence that a blood sacrifice would improve the odds of the system booting at this point.

A few more things to think about.

The kernel in tar.bz2 form is 75MB. Expanded, the kernel is 505MB. Compiled, the kernel is 853MB. Huh? One would expect a much larger size of the binaries given the size of the source. Of course, not all of that code gets compiled. Just the arch directory is 117MB, of which only 10MB is for x86. There's at least 30MB of drivers and filesystems I disabled. Still, that's like a 3-to-1 size ratio. I made a quick "Hello, World!" style program and the binary came in at over 60 times the size of the source file. Suffice it to say that I really have no idea what is going on in the kernel source code or build process.

The build time can vary greatly. Just hit make and walk away. Even better, start it before you go to bed or leave for work or some other time when you can just ignore it, rather than go through the whole "watched pot never boils" pain.

Don't use an important production system to try this out on. This a great time to use a virtual machine, be it VirtualBox, VirtualPC, VMWare, Parallels etc.

It's unlikely that you'll be able to significantly impact your system, at least in a positive way. Rendering your machine unable to boot is pretty easy, but getting any actual benefit out of a custom kernel is tough.

Start with a working .config. I know I said that already. It's worth repeating.

Be careful with the boot loaders. A "make all" will run an install script for LILO that will wreck your world if you're using GRUB.

Don't try to go it alone. There are lots of resources out there. In particular, different distros will have specific instructions for anything odd.

Resources

A bit like an Academy Award speech, except that I'm genuinely grateful. My thanks to the anonymous creators of the resources I used; I really really couldn't have done it without them.

Kernel.org, of course, and the README that comes in the Linux kernel tarball. This was the only place that recommended the "O=/path/to/build_dir" option. However, it somewhat dubiously says that LILO is the most likely boot loader.