OSCON 3.3: Current State of the Linux Kernel

Greg managed to fit a lot about historical issues surrounding release engineering and source control for Linux in his 45 minutes, and still had time to explain the various solutions the kernel team has tried. I'm going to just give the highlights of the issues before relating the current ways that the kernel team is trying to fix these issues.

Up until 2.5.3, Linus was accepting all changes to the Linux kernel only as emailed patch files, and eventually the poor scaling of this came to a head. At that point, he switched to using BitKeeper, a proprietary distributed revision control system, and while the license made people crazy, processes improved significantly, making kernel life generally better for a while. But the license issue would come back to haunt them when BitKeeper changed the license in a way that the kernel team could no longer work with.

At the same time, release engineering was pretty poor, and as the time between successive releases began to get longer and longer in the 2.6 timeframe, end users and distributions started taking exception to the wait (especially for security fixes).

Over the past few months, the kernel team has implemented several new tools and processes to deal with the outstanding problems.

First, a bug fix patch-only tree was created that had very strict requirements for accepted patches. This was used to produce 2.6.x.y patch releases to fix critical issues only.
Each .y patch series is dropped and recopied from the mainline with each .x release. Each .y series usually contains around three orders of magnitude fewer changes than are in a .x release, making people feel very safe in following .y patch releases.

To deal with the problem that .x "release candidates" were nothing of the sort, a new policy was created recently that after each .x release, a one-week window is opened for changes (the "patch flood"). After that week, rc1 is released, and really is a release candidate. After that point, all accepted patches must be bug fixes only, with no new or changed functionality. Once bug fixes die down, a new .x release is done, and the cycle repeats.

Finally, when the BitKeeper license changed, Linus and crew were left holding the bag, so they investigated the available options and found them all lacking. There was only one thing to do -- they created their own. More than one, actually. The main one right now is git, but some subsystems are using Mercurial instead. Greg mentioned others, but I have forgotten them.

These new SCMs share two major attributes -- they are lightning fast at importing patches, and they are distributed by design. They are also all very young, and Linus has said that in another three months he will see how far each project has gotten, and choose a new one for himself (and presumeably his core team).

Switching to a whiz-bang mode, Greg talked about a number of cool things recently merged or soon to be merged into the mainline kernel, including the Xen virtualization technology, lots of new file systems, improved internal APIs, and so on. He also proudly announced that Linux now supports more devices on more platforms than any other operating system ever (Linux passed NetBSD last year, an impressive achievement). In fact, there are now a number of operating systems that directly use Linux drivers so that they won't have to recreate the whole driver corpus.

Finally, he talked about stability of APIs. Internal kernel APIs are never going to be stable, but external APIs should remain so -- though he admitted that this only applied to syscalls, not to sysfs and procfs; stability of the latter is a subject of discussion these days. He pleaded with vendors to get out-of-tree drivers into the mainline so that they can be magically fixed every time internal APIs change, and pointed people to the stable_api_nonsense.txt file in the tree for more details.

I asked if these changes meant that a 2.8 series may never come, and he said that the new processes were forcing developers to do a much better job, no longer ripping out and replacing humongous chunks of code, but rather incrementally improving things until each major change was completed. They are discovering that they may not ever really need a new "pure development" kernel series, just more happy 2.6.x releases for years to come.

As an end user, that's just fine with me.

Geoff Broadwell
lives not far from O'Reilly headquarters in Santa Rosa, California, with a wonderful wife and daughter and four extremely spoiled cats. Geoff happily calls Perl the only computer language he ever really loved, having sampled a fair number before and since. He is on a personal mission to prove that dynamic languages are by far the best programming option for almost every purpose, and believes that the ultimate Linux distro of the future will contain little more than a kernel, an OpenGL and X server, the Parrot VM, and many, many Perl scripts.

Weblog authors are solely responsible for the content
and accuracy of their weblogs, including opinions they
express, and O'Reilly Media, Inc., disclaims any and
all liabililty for that content, its accuracy, and
opinions it may contain.