Ubuntu Server: Considering Kernel Configuration

Last week
we looked at Ubuntu Server's documentation, discussed hardware requirements, tried to figure out
what sets Ubuntu Server apart from Ubuntu Desktop, and what's included in the current release,
7.10 (Gutsy Gibbon). Ubuntu itself is not very helpful with these things, and I don't like to
write reviews that complain without also offering some answers, so today we're going to learn
how to dig into Ubuntu without installing it and find out these things for ourselves. We still
need the installation .iso image, but at least we won't have to install it or even burn
it to a CD to learn some useful information.

We're taking such a deep dive into the very bowels of Ubuntu Server that this is expanding
into a three-parter, so hold on to your hats and enjoy the ride.

What Is a "Server" Kernel?

We're going to compare the
/boot/config-2.6.22-14-server and /boot/config-2.6.22-14-generic files and find
out exactly what Ubuntu thinks a server kernel is, and what a desktop kernel is. We'll do this
by mounting the installation .iso in a temporary directory, and then extract and compare
the two files:

Check it out, we get both desktop and server kernels in one location, so this is going to be
easy. We want the last two, as those are the real kernel packages, which the linux-meta
packages are not. Copy them into separate directories for unpacking. .deb packages are
special archives that contain ordinary tar archives, and these two packages contain files
with the same names: control.tar.gz, data.tar.bz2, and debian-binary. So you
can't unpack them in the same directory.

Use the ar and tar commands to unpack the .debs, and then to unpack
their data.tar.gz files:

This gives us some nice manageable output — around 50 lines, rather than ~3100 lines
per file. So let's take a look at diff's output to see what's different.

I/O Scheduler

There are four different types of I/O scheduling: CFQ (Completely Fair Queuing), Deadline,
NOOP, and Anticipatory. Ubuntu makes CFQ the default for desktop kernels, and Deadline for
server kernels. The goal is the same for all of them: to optimize hard disk bandwidth for
different classes of workloads. In your configuration file this is the CONFIG_DEFAULT_IOSCHED
option, plus the CONFIG_IOSCHED_CFQ, _DEADLINE, _AS, and _NOOP options.

CFQ tries to balance all read/write requests equally.

Deadline gives a higher priority to read requests, and will re-order read/write
requests aggressively to meet the goal of completing read requests within a specified time,
without "starving" write requests, which are not given deadlines.

Anticipatory aims to reduce latency by giving priority to already-running
applications. It is supposed to be suitable for smaller systems with one or two hard disks, and
single or dual-core CPUs.

NOOP is a minimal scheduler for systems with hardware that handles I/O scheduling,
like large SCSI RAID arrays.

The question of which one is appropriate depends on your systems and how you use them: how
many CPUs, how many hard disks and controllers, what types of applications, and the loads your
systems have to handle. You can run benchmarks, and then tune your systems accordingly. You can
pass scheduler options in as boot-time options, or you can even enable different schedulers per
block device and change them on-the-fly (see Resources). The Ubuntu defaults are good starting
points, and if you need to tweak the settings they're just as tweakable as on any Linux.

Preemption

The server kernel has kernel preemption turned off
(CONFIG_PREEMPT_NONE=y), while the desktop kernel has it enabled (CONFIG_PREEMPT_BKL=y,
CONFIG_PREEMPT_VOLUNTARY=y). Preemption works along with scheduling to fine-tune performance,
efficiency, and responsiveness. In non-preemptive kernels, kernel code runs until completion;
the scheduler can't touch it until it's finished. But the Linux kernel allows tasks to be
interrupted at nearly any point (but not when it is unsafe, which is a whole huge fascinating
topic all by itself), so that tasks of lesser-priority can jump to the head of the line.

This is appropriate for desktop systems because users typically have several things going at
once: writing documents, playing music, Web surfing, downloading, and so on. Users don't care
how responsive background applications are, but only the ones they're actively using; so if
loading a Web page takes a little longer while the user is writing an email, it's an acceptable
trade-off. Overall efficiency and performance are actually reduced, but not in a way that annoys
the user.

On servers you want to minimize any and all performance hits, so turning off preemption is
usually the best practice.

Memory

The 32-bit server kernel supports up to 64 GB of memory; the desktop kernel,
a mere 4 GB (CONFIG_HIGHMEM64G=y, CONFIG_HIGHMEM4G=y). You'll only see these options in 32-bit
kernels because the 32-bit address space is big enough to support only 4 GB without trickery. Or
by using the Intel PAE (Physical Address Extension) mode, if you want to get technical. Linux
supports PAE, and you also need PAE support in your CPU. Anything newer than a Pentium Pro or
AMD K6-3 should be fine. On a 64-bit system you won't see any memory options because it doesn't
need hacks to overcome a lack of memory addressing space; you should be fine until your needs
exceed 16 exabytes of RAM.

Ticks and HZ

Both kernels support on-demand interrupt timers (CONFIG_NO_HZ=y), or
the so-called "tickless" option. This means that during periods of no activity, the system goes
into a truly idle state, which is supposed to save on power and cooling.

The server kernel is set to a timer interrupt rate of 100 Hz (CONFIG_HZ=100,
CONFIG_HZ_100=y), which means it accepts 100 interrupts per second. Another way to think of this
is the kernel looks up and peers around 100 times per second for something to do. The desktop
kernel is set to 250 Hz; lower numbers = lower overhead and higher latency, higher numbers =
higher overhead and lower latency. Higher numbers generally mean the system feels more
responsive, at the price of higher CPU usage. Some processes require more interrupts; for
example, video processing and VoIP servers need 1000 Hz. If you need to change the Hz value it
requires a kernel re-compile.

Come back next week to complete our kernel comparisons, and to finish our deep dive into the
guts of Ubuntu Server.