An Interview with Preemptible Kernel Patch Maintainer Rober

In this interview, Linux Journal chats with Robert Love, the principal maintainer
of an increasingly popular kernel-preemption patch that improves
the real-time responsiveness of the Linux kernel. Robert describes
his role in the project, explains why the preemption enhancement is
important to a broad range of Linux applications beyond just
embedded/real-time (including end-users' desktops) and shares his
vision of the future of Linux in the embedded and desktop
markets.

Robert Love, Principal Maintainer of the Preemptible Kernel Patch

Rick You have been becoming
well known as the maintainer of a preemptible kernel patch for the
Linux kernel. How did that situation come about?

Robert I am interested in
innovative ideas, and the preemptible kernel is a very neat
concept. The idea was first talked about during 2.3 (we finally had
a kernel that had fine-grained enough locking that it was
doable)--Linus actually outlined the first design. MontaVista
released a patch in this time frame and worked on it through the
early days of 2.4.

About this time I tried the patch, rediffed it to the newest
kernel and starting making changes. I eventually posted them.
People started trying it, and use of the patch took off. I've been
working hard since.

Rick What is different
between what you have done and what was (or is) being done by
MontaVista Software with their preemptible kernel patch? Are you
now collaborating with them, and if so, how closely?

Robert The patch I am
maintaining now is MontaVista's. They still put time into the
patch, and I work closely with a couple of their engineers.
Obviously it has evolved a lot since I got involved, but it's still
their work. I believe they are using the latest release in their
product.

Rick As I understand it,
your interest in the preemptible kernel is for reasons that would
tend to be of interest to a broader range of Linux applications
such as streaming of audio/video, etc., as opposed to the
embedded/real-time system focus of MontaVista. Is this accurate?
Can you give some examples of the sorts of applications that your
patch can noticeably improve the performance of?

Robert This is correct. I
see a preemptible kernel as a means to an overall better system.
Besides the traditional markets for low latency (audio/video,
specialized embedded/real-time, etc.), a preemptive kernel can
benefit any interactive task. The result is hopefully a smoother,
more responsive desktop.

Rick With the preemptible
kernel patch added, does Linux outperform what the latest versions
of MS Windows can offer to desktop, enterprise and embedded
users?

Robert I've got no idea how
we perform compared to Windows. I'd certainly say even earlier
versions of Linux were superior to Windows 9x. How we compare to
Windows NT (i.e., 2000 and XP) for desktop performance is hard to
measure.

The latency of the kernel certainly can be
measured—scheduling latency, jitter, etc., are quantitative—but
the quality of the desktop only begins with the kernel. That means
X and everything on top of it factors in, too.

Rick When Red Hat announced
that they were going to use RTLinux to provide real-time
capabilities to their customers, both Mike Tiemann (CTO of Red Hat)
and Alan Cox went on record to say that real time doesn't belong in
the kernel. Would you care to comment on that?

Robert Red Hat has an agenda
here, and I tend to disregard any technical comments when there is
bias. There are benefits to the hard real-time interrupt-driven
approach, I don't deny, but that solution is not Linux. I think
Linux can become a contender in the embedded/real-time market
without giving up on itself, while still being a UNIX and having
the standard Linux API. In fact, I think a lot of the technologies
that achieve this could live right in the official kernel. Kernel
preemption is one such innovation, and it's an innovation that does
not benefit solely real-time applications.

Rick Have you seen any
indication from Linus or other key kernel maintainers that your
patch (or the MontaVista patch) is being seriously considered for
the 2.5 kernel? What do you think the chances are that it will get
incorporated into 2.5 or 2.6?

Robert Linus said at ALS
[Annual Linux Showcase] this year [2001] that he was interested in
the preemptible kernel patch. That doesn't mean anything to me
until we are in, but it is a good sign. There is opposition. There
are various issues that need to be dealt with. I believe it is a
sane move for 2.5. The patch has seen a lot of testing, and we have
a lot of users. I do not want to predict whether it will be merged
for 2.5. Time will tell. [Subsequent to this interview, the
announcement was made that Linus has merged the patch into the main
Linux development-kernel tree, beginning with version
2.5.4-pre6.]

Rick Please summarize the
advantages in general, not just for embedded real-time
applications, of having the preemptible kernel enhancement included
in the kernel. What about any disadvantages?

Robert I'll start with a
quick explanation of how the patch works. Right now, the kernel is
not preemptible. This means that code running in the kernel runs
until completion, which is the source of our latency. Although
kernel code is well written and regulated, the net result is that
we effectively have an unbounded limit on how long we spend in the
kernel. Time spent in kernel mode can grow to many hundreds of
milliseconds. With some tasks demanding sub-5ms latencies, this
non-preemptibility is a problem.

The preemptible kernel patch changes all this. It makes the
kernel preemptible, just like user space. If a higher-priority task
becomes runnable, the preempt patch will allow it to run. Wherever
it is. We can preempt anywhere, subject to SMP- [symmetric
multiprocessing] locking constraints. That is, we use spin locks as
markers for regions of preemptibility. Of course, on UP
[uni-processing] they aren't actually spin locks, just
markers.

The improvement to response is clear: a high-priority task
can run as soon as it needs to. This is a requisite of real-time
computing, where you need your real-time task to run the moment it
becomes runnable. But the same effect applies to normal interactive
tasks: as soon as an event occurs (such as the user clicking the
mouse) that marks it runnable, it can run (subject to the
non-preemptible regions, of course).

There are some counter arguments. The first is that the
preemptible kernel lowers throughput because it introduces
complexity. Testing has showed, however, that it improves
throughput in nearly all situations. My hypothesis is that the same
quicker response to events that helps interactivity helps
throughput. When I/O data becomes available and a task can be
removed from a wait queue and continue doing I/O, the preemptible
kernel allows it to happen immediately—as soon as the interrupt
that set need_resched returns, in fact. This means better
multitasking.

There are other issues, too. We have to take care of per-CPU
variables, now. In an SMP kernel, per-CPU variables are implicitly
locked—they don't have explicit locks, but because they are unique
to each CPU, a task on another CPU can't touch them. Preemption
makes it an issue because a preempted task can trample on the
variables without locks.

Overall, I think the issues can be addressed, and we can have
a preemptible kernel as a proper solution to latency in the
kernel.

Rick From your brief
explanation of how the preemptible kernel does its business, I get
the impression that nothing new needs be done on a user's system
other than installing the modified kernel in order to gain
benefits. Specifically, if I were to install the patched kernel on
my desktop system, would I immediately begin noticing performance
improvements during my work (and play) on the system, such as doing
work during downloads or burning CDs while listening to streaming
audio or watching streaming video? Or, do I need to have
applications that can take advantage of the kernel's preemption
capability? Would it be necessary to alter the priority of the apps
I run, or make any other adjustments?

Robert You don't have to do
anything. The preemptible kernel patch requires no change in user
applications, environment or API. The change (from switching to a
preemptible kernel) should be immediate. Your normal programs and
your normal benchmarks will show latency improvements. Renicing
your programs would help, just because they would be able to reap
the benefits of preemption more readily (anything they need to do
could cause a preemption).

This is why I see elegance in this solution; Linux can be
made more RT without abandoning itself.

Rick Please describe
yourself, what you do and what your interests are. Also, what is
your background relative to Linux and your philosophy with respect
to open-source software?

Robert I am a student at the
University of Florida studying mathematics and computer science,
originally from South Florida. I first used Linux six or seven
years ago with an early Slackware release. I've run nothing but
Linux for the past couple years.

Outside of computing I'm really into rock music, good food
and the military-industrial complex.

My philosophy with respect to open source is that of a
pragmatist. While I enjoy the freedom of the code and certainly
appreciate the rights gained from free software, I see the biggest
benefit in merely having open code. I can see it; I can fix it; I
can play with it. With enough momentum, open source creates amazing
software. I am very proud of our kernel.

Rick Approximately how much
time per week do you spend working on your kernel patch for
Linux?

Robert My girlfriend would
probably say too much. Anywhere from a couple hours a week to many
hours a day.

Rick What is your vision of
the future of Linux in the embedded market and in the home and
business desktop computer market?

Robert Linux is going far in
the embedded market. I'm sure you as much as anyone can see the
vast potential there. Linux offers key benefits—particularly
available source and a highly configurable system—that should
appeal to the embedded space. It is light, so it embeds well. It
has a simple API. A lot of user-space solutions exist, and it is
easy to create entirely new ones.

I certainly think Linux is an excellent desktop system (both
GNOME and KDE are quite mature), but I don't see huge strides here
in the coming year. Market share will certainly increase, perhaps
significantly, but we will not be overtaking any evil monopolies
this year.

Editor's note: Look for an
article by Robert on his preemptive work in next month's
issue.

Rick Lehrbaum
(rick@linuxdevices.com)
created the
LinuxDevices.com and
DesktopLinux.com web
sites. Rick has worked in the field of embedded systems since 1979.
He cofounded Ampro Computers, founded the PC/104 Consortium and was
instrumental in creating and launching the Embedded Linux
Consortium.

Trending Topics

Upcoming Webinar

Getting Started with DevOps - Including New Data on IT Performance from Puppet Labs 2015 State of DevOps Report

August 27, 2015
12:00 PM CDT

DevOps represents a profound change from the way most IT departments have traditionally worked: from siloed teams and high-anxiety releases to everyone collaborating on uneventful and more frequent releases of higher-quality code. It doesn't matter how large or small an organization is, or even whether it's historically slow moving or risk averse — there are ways to adopt DevOps sanely, and get measurable results in just weeks.