Hacking the kernel: one man's tale

Mel Gorman works with code. Nothing unusual in that, an overwhelming number of the people I interact with in the FOSS arena do just that.

But Gorman (pictured above) is a little more involved than most: he is a kernel developer and hence works at a more basic level than many others.

Gorman is based in Ireland from where he works remotely for SUSE, the Linux company based in Nuremberg, Germany. He is involved both in work for the company and with the upstream kernel community.

In his early 30s, Gorman is not a superstar in the kernel ranks; he is one of the rank and file who contributes to a project that keeps a fair amount of the big iron in the world running. But these are the worker bees of the FOSS community, the silent majority who keep the codebase churning over.

iTWire spoke to him at length to find out how an average kernel developer takes up the job, what he or she actually does, and what keeps them going.

Was there some influence from your parents, a relative or a friend that made you decide to study computer science?

I felt this had to be answered while lying on a couch. My childhood was not particularly technically oriented. I grew up in a rural location in 1980s Ireland. Computers were not common and I do not have much of a reference point to compare with. Maybe my childhood was quite technical to most people and I did not realise it.

There was a C64 in the house that belonged to my older brother as he did have a very keen interest that I guess rubbed off on the occasions he was home from boarding school. Games dominated my computer activity and while I took some interest in programming in Basic and C64 assembler, I do not remember having a particularly deep understanding. I knew some tricks like using the raster interrupt to generate pretty patterns but it was more cargo cult programming based on manuals and magazines than any real understanding. Reading a C64 manual at eight stretches the reading comprehension skills a bit.

Based on that exposure I did decide at 11 that I wanted to work with computers in some capacity so I did find computers sufficiently interesting to make the basics of a career choice at a young age. My decisions in secondary school supported that early choice but as I was in a boarding school with limited technical resources I did not get much access to computers again until I reached university.

When did you first get interested in free and open source software?

I had a Linux shell account when I was in first year but I did not even know what Linux was at the time. It was the latter half of second year before I took an interest in free and open source software. It was encouraged by other members of the computer society.

Is it an ideological thing or are you more of a pragmatic person in your approach to FOSS?

It was a mix of both. From a pragmatic point of view I was in university to learn computer science and software engineering. Even at that point I found that it was very difficult to figure out how proprietary software was put together. The interesting details were simply unavailable and my perception was that complex application development depended on cobbling together many closed-source toolkits and widgets and working around any oddities. If off-the-shelf components did not work exactly as required, then tough luck. This was drastically different to what I initially experienced with the C64 where it was always possible to poke around how it worked.

When I started working on proprietary software I found that sometimes this attitude still persisted even when the source was available. Poking into other teams' software components had a tendency to, at best, have the emails ignored and/or any changes backed out from source control. This depends on the company but I only worked on one company where this was not the case. I do not know if that still happens today but I have no reason to believe that anything has changed.

Free and open source software was different. The details were always available and to varying degrees developers would discuss different aspects of it. Sometimes the enthusiasm bordered on the manic which could be off-putting but it was always interesting. This did not mean that the software was better. In many cases it was worse that the proprietary alternative, although that has improved drastically over the years. The important thing was that the opportunity was available to learn how it worked and improve it. From a learning perspective, this makes sense and I find that I'm still learning.

Unfortunately, my early career was dominated by proprietary software work. I had not put in the necessary time on FOSS during University to base a career on it.

You were a teacher for a while before you started working in a corporate setting. Which job do you prefer and why?

Actually my road is a little more complex. I went to University and had two summer jobs while there, one proprietary and the other internal-only code. On graduation, I worked on proprietary software for a year before going back to do a masters in which software development work was exclusively FOSS because this is what I found interesting. I was teaching for six months after I completed my masters, but then had to bite the bullet and take another proprietary development job for a few years. I then started a part-time PhD which allowed me to work three days proprietary and two days FOSS a week which gradually became full-time working on FOSS.

I prefer working on software to teaching it although six months is not much of a comparison. I fear that if I was teaching exclusively that I would get frustrated revisiting the exact same topic each year and working with people who do not necessarily want to be there. I know this is not the reality as technology and teaching methods change each year but I do not think it is for me. I might revisit the decision in the future.

Your first kernel-related job was in January 2006 at IBM Linux Technology Centre. Can you detail the steps that led up to this?

I was working part-time on my PhD for a year with IBM when I was contacted by the LTC asking if two of my email addresses
This email address is being protected from spambots. You need JavaScript enabled to view it.
the same person. The kernel team had a number of long-term development goals and my research had the potential to help deliver them. They were interested in hearing what I did within IBM and how long it would take me to complete the research. Martin Bligh arranged an internal transfer with help from Patricia Gaughen, Andy Whitcroft and Dave Hansen so I could work on the research full-time for which I was very grateful. On one hand, I suppose I am lucky but it took a long time to get to the point where I could work on FOSS.

What exactly do you do at SUSE?

My most important role is as the last level of support for SLES customers. Others have the same roles and we are responsible for developing patches or making tuning recommendations to address any bugs reported against the SLES kernel that the support people cannot deal with. Primarily I deal with virtual memory management problems but not exclusively so.

Any bug fixes that are developed for customer bugs are pushed to mainline and backported to -stable where possible. In this way any functional fix I add to the SLES kernel will be removed later when it gets backported to the 3.0-stable series. Some performance works also make it through but it is relatively rare I push complex performance patches through -stable.

To preempt bugs being filed I automate tests heavily and monitor SLES kernels and mainline kernel performance over time. I watch out for the type of regressions that have resulted in customer bug reports in the past and preemptively address them as it can take a weeks to solve the more complex issues and merge them upstream. This minimises the number of bug reports filed but also means in a number of cases where a bug is failed that we have candidate fixes already available. As openSUSE uses mainline kernels the same work indirectly helps them.

If a project is likely to take a considerable amount of time, I'll file a feature request so project management is aware and also so they can priorities if it is not possible to deliver on all of the projects in a reasonable timeframe.

Most of the long-lived development work happens in mainline first and is then backported. I review a number of other people's patches but tend to pay more attention to those that I think I'd want to backport to SLES.

Are you working on the kernel only for SUSE, or are you also working with the kernel devs who work in tandem with Linus?

I work with both groups and am active in the upstream kernel community. I was also program committee member for the last Linux Storage, Filesystems and Memory Management summit and for the Kernel Summit. The SLES kernel policy is "upstream first" which means that in almost all cases the patches in SLES are backports from the mainline kernel. There may be times when patches that have not reached a mainline release yet are in the SLES kernels but this is a temporary situation. The happy result of this is that most of us are active upstream and that most patches in SLES have received external community review.

How does one go about getting patches accepted?

In my particular case it is a relatively straight-forward process. The details depend on the exact circumstance but broadly speaking patches are fixing a functional bug, fixing a performance bug or adding a new feature. There are typically three sets of people involved other than the patch author - the bug reporter (if they exist) or independent testers, the community reviewers and the upstream maintainer. The basic workflow is essentially an iterative development model involving all parties.

If a bug reporter exists then I send them candidate fixes until there is a final version that they report fix the problem. Community reviewers involved in the same area will either add their Acked-by/Reviewed-by or suggest better alternatives. I repeat until both the bug reporter and reviewers are happy and then send that to the area maintainer (almost always Andrew Morton in my case) with all interested parties cc'd. The information in such a patch's changelog will describe what the problem was, what the user-visible impact is and how the patch fixes it. The user-visible impact is important for helping maintainers decide if the patch should be added to the -stable tree or not.

For performance bugs or new features the details are slightly different. This is usually a series of patches and the leader is expected to describe the problem and a short summary of what the series does to address it. In almost all cases I'll include details on how the impact of the series is measured and details of any benchmarks used to validate the patch series. This usually goes through a few rounds of review and when everyone is happy, it's resent to Andrew with everyone cc'd.

It is not unusual for a patch or a patch series to go through a number of revisions before it is finally accepted.

Do you like what you are doing? Do you feel that it improves your perspective on things, or does it drag you down?

Yes, I like what I'm doing. As with everything, there are good and bad days but in general I like what I'm doing. I'm not sure how much it changes my perspective in terms of working. The primary downside is that I work from home and there are no similar developers living nearby that I know of. Hence, this particular job can be quite isolating but I balance it out with non-work related activities. On the plus side, there are relatively few meetings.

FREE WHITEPAPER - RISKS OF MOVING DATABASES TO VMWARE

VMware changed the rules about the server resources required to keep a database responding

It's now more difficult for DBAs to see interaction between the database and server resources

This whitepaper highlights the key differences between performance management between physical and virtual servers, and maps out the five most common trouble spots when moving production databases to VMware

A professional journalist with decades of experience, Sam for nine years used DOS and then Windows, which led him to start experimenting with GNU/Linux in 1998. Since then he has written widely about the use of both free and open source software, and the people behind the code. His personal blog is titled Irregular Expression.