1.†
Introduction

In August 2006, a large part of the realtime-preempt kernel patch that
is maintained by Ingo Molnar, Thmomas Gleixner and Steven Rostedt was merged
into the mainline kernel and is now immediately available. A number of functions,
however, still is only available, if the realtime-preempt patch is applied.
It is planned to merge the vast majority of these functions into the mainline
kernel before the end of 2008.

In addition to the traditional installation method that requires download,
patch and recompilation of the kernel, an apt repository is now provided
that greatly facilitates the installation of the realtime-preempt patch. This
mechanism, however, is currently only available when the Ubuntu Feisty
distribution is used. It is expected that Redhat Enterprise Linux and
Fedora will contain prepatched realtime-preempt kernels in the near future.

This HOWTO explains how to install, configure, test and use the realtime-
preempt kernel patch in Ubuntu Feisty (7.04) and other 2.6 systems.

2.†
Installation

2.1.†
Ubuntu Feisty (7.04)

Add, as root, to your /etc/apt/sources.list (remember to do a backup of this file) the following line:

These settings, however, are only useful during development and evaluation.
Some of them are better disabled when the system goes into production.
The scheduling latency and the missed timer offsets histograms,
however, only slightly interfere with the system and may, thus, be used
to continuously register the wakeup latency under production conditions.
In any case, make sure that stack overflow checking is disabled, since this
may produce additional latencies when enabled:

Kernel hacking --->
[ ] Check for stack overflows

The new kernel is then compiled, linked and installed as usual.

# make
# make modules_install install

When the system is rebooted, the newly provided kernel becomes part of
the boot menu and can be selected.

Should you happen to be the proud owner of a multi-core processor, be
sure to specify the -j <jobs> option of
make where <jobs> is
twice the number of cores your processor has, as this will speed up
kernel compilation considerably.

3.†
Testing and using the realtime-preempt patch

3.1.†
Built-in tools

The kernel version must now contain the tags PREEMPT
and RT such as

uname -v
#42 SMP PREEMPT RT Fri Nov 6 18:55:29 CET 2009

or something went completely wrong, otherwise.

In order to access the built-in diagnostic tools, the debug file system must be
mounted. This is either done manually by entering

to the file /etc/fstab to have the debug file system mounted
automatically at boot time.

Note

In earlier kernels, the built-in diagnostic tools were immediately available in
the proc file system that is normally mounted by default.

The histograms of the wakeup latencies (one per CPU) are available here:

# ls /sys/kernel/debug/tracing/latency_hist/wakeup_latency/CPU?

From 2.6.2.31 onwards, the files are in:

# ls /sys/kernel/debug/tracing/latency_hist/wakeup/CPU?

This configuration feature is always available in the prebuilt Ubuntu
kernels; in the locally created kernels it is only available, if configured
as shown above. The granularity of the histograms amounts to one microsecond:

Some more documentation is available in the kernel source directory
Documentation/trace/histograms.txt.

3.2.†
External testing tool

In addition, Thomas Gleixner made available the test tool cyclictest, that allows
to better and more spcifically determine the realtime capabilities of a given system. The sources
can be downloaded from here.
If you installed the Ubuntu apt packages (see above), the cyclictest tool is
available immediately. The git repository
contains other useful test programs for real-time systems.

Running cyclictest only over a short period of time and without creating
appropriate real-time stress conditions is rather meaningless, since the execution of an
asynchronous event from idle state is normally always quite fast, and every - even non-RT
system - can do that. The challenge is to minimize the latency when reacting to an
asynchronuous event, irrespective of what code path is executed at the time when the
external event arrives. Therefore, specific stress conditions must be present while
cyclictest is running to reliably determine the worst-case latency
of a given system.

3.3.†
Latency fighting

If - as in the above example - a low worst-case latency is measured, and this is the case
even under a system load that is equivalent to the load expected under production conditions,
everything is alright. Of course, the measurement must last suffciently long, preferably
24 hours or more to run several hundred million test threads. If possible, the
-i command line option (thread interval) should be used to increase the
number of test threads over time. As a role of thumb, the thread interval should be set to
a value twice as long as the expected worst-case latency. If at the end of such a test period
the worst-cae latency still did not exceed the value that is assumed critical for a given system,
the particular kernel in combination with the hardware in use can then probably be regarded
as real-time capable.

What, however, if the latency is higher than acceptable? Then, the famous "latency fighting" begins.
For this purpose, the cyclictest tool provides the -b option
that causes a function tracing to be written to /sys/kernel/debug/tracing/trace,
if a specified latency threshold was exceeded, for example:

# ./cyclictest -a -t -n -p99 -f -b100

This causes the program to abort execution, if the latency value exceeds 100 microseconds; the
culprit can then be found in the trace output at /sys/kernel/debug/tracing/trace.
The kernel function that was executed just before a latency of more than 100 microseconds was
detected is marked with an exclamation mark such as

The first column indicates the calling process responsible for triggering the latency.

If the trace output is not obvious, it can be submitted to the OSADL Latency Fight Support Service at
<latency-fightersªosadl.org>
In addition to the output of cat
/sys/kernel/debug/tracing/trace, the output of lspci and the .config
file that was used to build the kernel in question must be submitted. We are sure you understand that
OSADL members will be served first, but we promise to do our best to help everybody to
successfully fight against kernel and driver latencies.