Real Time and Linux

What is real time? This article, first of a three-part series, introduces the benchmarks we'll run on realt-ime Linux version in the next two issues.

Is Linux Capable of Real Time?

Unless we state otherwise, assume we are talking about the
2.4.9 version of the Linux kernel. Version 2.4.9 was released in
August 2001, although our statements, at least for the most part,
are true for the last several years of kernel releases.

There are many qualities of an operating system that may be
necessary or desirable for it to be appropriate for real-time
applications. One list of features is included in the Comp.realtime
FAQ at
http://www.faqs.org/faqs/realtime-computing/faq/.
That list contains such features as the OS being multithreaded and
preemptible, and able to support thread priorities and provide
predictable thread-synchronization mechanisms. Linux is certainly
multithreaded, supports thread priorities and provides predictable
thread-synchronization mechanisms. The Linux kernel is not
preemptible.

The FAQ also says that one should know the OS behavior for
interrupt latency, time for system calls and maximum time that
interrupts are masked. Further, one should know the
system-interrupt levels and device-driver IRQ (interrupt request
line) levels, as well as the maximum time they take. We provide
some timings for interrupt latency and interrupt masked
("interrupts blocked'') times in the benchmarking section
below.

Many developers also are interested in having a deadline
scheduler, a kernel that is preemptible in the millisecond range or
better, more than 100 priority levels, user-space support for
interrupt handlers and DMA, priority inheritance on synchronization
mechanisms, microsecond timer resolution, the complete set of POSIX
1003.1b functionality and constant time algorithms for scheduling,
exit(), etc. None of these capabilities are available in the
standard kernel and the GNU C library.

Additionally, in practice, the magnitude of delays becomes
important. The Linux kernel, in a relatively easily constrained
environment, may be capable of worst-case response times of about
50ms, with the average being just a few milliseconds.

Some applications, however, may require response times on the
order of 25µs. Such requirements are not satisfiable in
applications that are making use of Linux kernel functionality. In
such cases, some mechanism outside of the Linux kernel functions
must be employed in order to assure such relatively short
response-time deadlines.

We see, in practice, that a hard real-time operating system
can assure deterministic response and provide response times that
are significantly faster than those provided by general-purpose
operating systems like Linux.

We see that Linux is not a real-time operating system because
it always cannot assure deterministic performance and because its
average and worst-case timing behavior is far worse than that
required by many real-time applications. Remember that the timing
behavior required by these many real-time applications generally is
not a hardware limitation. For example, the Linux kernel response
time may be on the order of a few milliseconds on a typical
x86-based PC, while the same hardware may be capable of better than
20µs response times when running a real-time operating
system.

The two reasons that the Linux kernel has such relatively
poor performance on uniprocessor systems are because the kernel
disables interrupts and because the kernel is not suitably
preemptible. If interrupts are disabled, the system is not capable
of responding to an incoming interrupt. The longer that interrupts
are delayed, the longer the expected delay for an application's
response to an interrupt. The lack of kernel preemptibility means
that the kernel does not preempt itself, such as in a system call
for a lower-priority process, in order to switch to a
higher-priority process that has just been awakened. This may cause
significant delay. On SMP systems, the Linux kernel also employs
locks and semaphores that will cause delays.

Real-Time Application Programming

User-space real-time applications require services of the
Linux kernel. Such services, among other things, provide
scheduling, interprocess communication and performance improvement.
Let's examine a variety of system calls (the kernel's way of
providing services to applications that are of special benefit to
real-time application developers). These calls are used to
constrain the operating environment.

There are 208 system calls in the Linux kernel. System calls
usually are called indirectly through library routines. The library
routines usually have the same name as the system call, and
sometimes library routines map into alternative system calls. For
example, on Linux, the signal library routine from the GNU C
library, version 2.2.3, maps to the sigaction system call.

A real-time application may call nearly all of the set of
system calls. The calls that are most interesting to us are
exit(2), fork(2), exec(2), kill(2), pipe(2), brk(2),
getrususage(2), mmap(2), setitimer(2), ipc(2) (in the form of
semget(), shmget() and msgget()), clone(), mlockall(2) and
sched_setscheduler(2). Most of these are described well in either
Advanced Programming in the UNIX Environment,
by W. Richard Stevens, or in POSIX.4: Programming for the
Real World, by Bill O. Gallmeister. The clone() function
is Linux-specific. The others, for the most part, are compatible
with typical UNIX systems. However, read the man pages because
there are some subtle differences at times.

Real-time applications on Linux also frequently are
interested in the POSIX Threads calls, such as pthread_create() and
pthread_mutex_lock(). Several implementations of these functions
exist for Linux. The most commonly available of these is provided
by the GNU C library. These so-called LinuxThreads are based on the
clone() system call and are scheduled by the Linux scheduler. Some
POSIX functions are available for POSIX Threads (e.g., sem_wait())
but not for Linux processes.

An application running on Linux ordinarily can be slowed down
considerably, from its best case, by a number of factors.
Essentially these are caused by contention for resources. Such
resources include synchronization primitives, main memory, the CPU,
a bus, the CPU cache and interrupt handling.

An application can reduce its resource contention for these
resources in a number of ways. For synchronization mechanisms,
e.g., mutexes and semaphores, an application can reduce their use,
employ priority inheritance versions, employ relatively fast
implementations, reduce the time in critical sections, etc.
Contention for the CPU is affected by priorities. In this view, for
example, lack of kernel preemption can be seen as a priority
inversion. Contention for a bus is probably not significantly long
to be of direct concern. However, know your hardware. Do you have a
clock that takes 70µs to respond and holds the bus?
Contention for a cache is affected by frequent context switches and
by large or random data or instruction references.

we've had terrible experience in particular versions of linux (i'd rather not specify which here), where a 'cp' running on the background for a few GB causes another process running a write() on the same partition to wait 5-20 seconds!. move from CFQ to deadline changed the formula a bit (to about 1.5 seconds to 8 seconds)!. I'm talking 8GB, 2 cpu, 3 GHz, Xeons (HT) running SCSI, and just 2 user processes (1 cp + 1 write()). talk about real-time linux. This is repeatable on any similar config.