Graybeards: Are Kernel Developers Becoming Extinct?

Each year the Linux Foundation invites selected kernel developers to the Collaboration Summit in San Francisco. More and more of their beards are turning gray.

The barbate characteristics of Richard Stallman, Alan Cox, John "Maddog" Hall and other forefathers of Linux and open source have long been iconic for hackers the world over. But as time grays the beards of these open source luminaries, the question becomes, "Who will replace them?" Just this question was posed by LWN editor Jonathan Corbet on April 14 during his roundtable talk at the Collaboration Summit of the Linux Foundation -- and opened a hornet's nest in the process.

Greg Kroah-Hartman, who works on the kernel for Novell, agrees that the old guard is still very much present, but needs to be because of the high change rate in the kernel. He added, "If we're in the way, let us know." Other developers concurred, that the code is complicated, but the oversight necessary. Only Andrew Morton showed signs of fatigue: "Yes, we're getting older, and we're getting more tired. I don't see people jumping with enthusiasm to work on things the way that I used to."

InformationWeek responded with a fitting title for its article on the Collaboration Summit: "Linux Graybeards? Yes, But Also a Wisdom Circle."

Is open source boring?

The jfplayhouse.com website went a step further by claiming that it wasn't a surprise that younger developers don't want to participate in the kernel. After all, they asserted, "Linux is one of the most boring open source software projects in existence. It's a massive heap of code that is mostly worked on by corporate developers." Add to that, that most kernel work is closed and there isn't very much interesting work to do on it: "Where is the appeal in spending your free time on a webcam driver when the completion of that driver will have zero bearing on the success of Linux?"

Comments to both articles included those from Jon Corbet himself, who opened up the subject in his Collaboration Summit talk in the first place. In Corbet's view kernel development is by no means dying off. On the contrary, he says, of the thousand or so developers involved in each three-month kernel release cycle, "nearly 20% of these contributors are working on their own time, because they want to." Among them are many first-time contributors. Corbet claims that the kernel is as attractive as ever, that there is more and more code to absorb into it. The issue, he says, is not the kernel itself, but the "graying of the ranks." Not that it is a problem. He continues, "James Bottomley suggested they may stay in place until they start dying off."

Graybeards as icons of wisdom

Sam Ramji, one-time open source representative at Microsoft, hosted an interesting panel at the 4th Collaboration Summit. Under the title "Does Open Source Mean Open Cloud?" Ramji called attention to his estimation that about 90% of all virtual machines run Linux, but that the SaaS structure supporting them is largely proprietary.

Therefore, the kernel and Linux community can continue to make use of graybeard services. This was also corroborated by Dan Frye of IBM in his keynote "10+ Years of Linux at IBM." When the company announced a decade ago that they would invest a billion dollars in Linux, it didn't fully understand the development model and tried in vain to bring code into the kernel. As it turned out later, both sides made some mistakes. Matt Mackall of the Mercurial project can remember how large blocks of code for hard disk volume management were rejected: "Linus didn't know what volume management was. For a long time it appeared to be going in, then it was rejected as 'just not salvageable'."

Meanwhile problems such as this have been dealt with more effectively and IBM has learned over the years how to best collaborate on kernel development. Google has also picked up some similar experience. After developing Android for quite a while behind closed doors and not coordinating with the kernel, new parts of Android are now finding their way back in. Kernel developers will now probably be confronted by huge new blocks of code. To prepare for the eventuality and contribute a positive atmosphere, Google handed out a Nexus One to each of the Collaboration Summit participants.

Related content

The 2011 End User Summit program was announced by the Linux Foundation. The End User Summit which is sponsored by IBM and Softlayer, is invitation only and designed to bring together enterprise Linux end users and kernel developers. Find who should be attending and how individuals can request an invitation.

The second day of the Linux Plumbers Conference got off to a productive start with an address by Jonathan Corbet, co-founder of LWN.net. "The Kernel Report" covered the challenges facing the kernel development community and the current or pending responses to those challenges, as well as a short summary of the Kernel Summit held earlier this week.

In the second keynote of the Linux-Kongress in Hamburg, Germany, cofounder of LWN.net and kernel developer Jonathan Corbet presented details on yesterday's released Kernel 2.6.27, but also described some of the work Linus Torvalds and his group of hackers have been up to.

Comments

Greybeards

Jon "maddog" Hall

I found this article interesting, particularly when mentioned with respect to kernel development, since my only direct conversation about the kernel and what it should do was many years ago with Linus about making it 64-bit and porting it to another platform to keep it portable and avoid "Intelisms".

Yes, I taught operating system and compiler design many years ago, and yes, I did system level programming in assembly language, but my "C" looks a lot like "Fortran" and I now leave system level coding up to others.

I have, over the years, encouraged the younger developers to talk to the "greybeards", because a lot of the problems we have encountered in Linux were solved by some of the "greybeards" in Unix systems...or at least we can help you avoid the mistakes we made.

I think, however, that kernel level development is very exciting, since it is in the kernel that one mistake can make the entire system crash....one routine can make your soft real-time unreliable. The comment that "no one cares about your web-cam driver I think is patently untrue. If it is difficult for young developers to get into "kernel development", perhaps it is because of the high barrier to entry....the need for the developer to be a good one. To fit their code into the already existing code base.

Many years ago a friend of mine named Professor John Lions (University of New South Wales) documented how the Unix system kernel was created, because he believed that the best way of teaching CS students how to write good code was to show them code written by good programmers. Unfortunately his book was not published until 20 years after he wrote it due to changes in AT&T licensing, but reportedly preliminary copies of it made it the most photocopied technical book of all time, and second only to the Bible in photocopies over-all. Students want to know how the system works.

When I learned CS in 1973 (I actually started in 1969, but there was little "Computer Science" at that time, only "computer black magic&quot we often created small "throw away" projects. We knew they would have no use after we were finished, so we did not put much time into them.

I now encourage educators to use FOSS code (and not just Linux, but BSD, FreeDOS, HURD and others) in teaching courses like operating system design, compiler theory and database design. But more importantly students in these classes can make incremental improvements to the projects if they realize that their code has to be integrated and maintained in the future by others. These are the real lessons of FOSS.....and yes, the "greybeards" have been doing this for forty years (or more).

How can someone (or their potential employer) not appreciate the fact that their code may be used by millions of people inside the Linux kernel (or the GNU suite of compilers, or the FOSS databases)?

I do not see a lack of younger programmers who would not be willing to take up the call, but they may need some "greybeards" to help them learn the way.