If anyone knows Linux kernel driverdevelopment, it's Greg Kroah-Hartman, who'sbeen working deep in Linux for over a decade. In this interview, Gregtalks about how the Linux project has accommodated the accelerating rate ofchange for the kernel, and offers some insight on where Linux is headed.Scott Swigart: Can you please introduce yourself and your association withLinux?

Greg Kroah-Hartman: Sure. I’m a Novell fellow, and I’m working on the Linuxkernel with the SUSE Labs division at Novell.

I’ve been doing Linux kernel workfor more than 10 years now. I’m a maintainer of several different subsystems,including USB, the driver core, and some other minor things. I’m alsoresponsible for releasing the stable Linux kernels. I used to be a maintainerof a lot of other driver subsystems and such over the years, but I’vethankfully handed them off to other people.

Scott: Let me start with a really big-picture question, sinceyou’ve been involved in Linux for a long time. Obviously, there’s still lots ofstuff happening on the mailing list, but fundamentally, the way code makes itinto the kernel hasn’t really changed, and yet the scale of development is muchlarger than it was, say, five years ago.

How have you seen Linux kerneldevelopment change to handle the scale of the effort that’s going on today?

Greg: I’ve been tracking this for over five years now, and we’vegotten a lot better. There are a lot more people doing the work.

To give an example, for the 2.5 to2.6 kernel development series, which took about two years, the top 30 peopledid 80 percent of the work. Now, the top 30 people do 30 percent of the work.The sheer number of developers has also increased. We were running a couplehundred developers, and now we’re running a couple thousand.

Scott: How does that get managed? Are there more layers in thehierarchy?

Greg: We’ve invented some tools to help us out a lot. It used tobe that Linus was the only one who could commit anything, or who would acceptanything. It was all done through email.

And then BitKeeper was developed,and a number of us started using that. It was very nice, because we had a muchfaster feedback loop with Linus. I could have him pull 50 patches, see that hepulled them, and then I could hit him with another 50 patches, if I wanted to,within a day. Before, we had a lag of a couple of weeks.

Then, when we couldn’t use BitKeeperanymore, Linus wrote Git, and then that just increased it even more, becauseeverybody can use Git. Our development model increased its capacity, and partof that was that we started trusting more people. Now we have subsystemmaintainers and such.

As I said, I maintain the subsystemssuch as USB, and I have people who I trust enough that if they send me a patch,I’ll take it, no questions asked. Because the most important thing is I knowthat they will still be around in case there’s a problem with it. [laughs]

And then I send stuff off to Linus.So, Linus trusts 10 to 15 people, and I trust 10 to 15 people. And I’m one ofthe subsystem maintainers. So, it’s a big, giant web of trust helping this goon.

We’ve developed some procedures thathelp guide how and when we do merges and releases. And we’ve a very regularrelease schedule: every three months, we do a release. We have a two-week mergewindow, and all those patches in that merge window have to have been tested.We’ve gotten the development process down really well over the past four years.

We’re also increasing the rate ofchange in our development. The same amount of work one of the top 10 developersdid last year wouldn’t have even made it into the top 20 this year. Ourindividual developers have got the work flow down, so we can actuallycontribute more, to an extent that’s amazing.

Scott: I understand that one of the things you’re involved with isthe Linux Driver Project. It seems like there was a certain tension betweenLinux and certain hardware vendors, in the sense that, for a long time,hardware vendors had been really used to not shipping driver code.

They were used to shipping forWindows, and they didn’t have to be involved with drivers. They just providedan installer. And then, as Linux started to become a lot more popular in theserver space and make some inroads in the desktop space, they wanted access tothat market.

There was kind of a culture shift ina lot of companies to realize how to provide things like drivers in a way thatmatched with what the Linux community expected. Take the example of somehypothetical Acme hardware vendor that wants to submit a driver but has neverdone it before. What does the process look like?

Greg: First, people need to realize that our driver model isdifferent than other operating systems, because all our drivers ship with thekernel. The license requires our drivers to be open, so everything is in themain kernel tree.

Because of our huge rate of change,they pretty much have to be in the kernel tree. Otherwise, keeping a driveroutside the kernel is technically a very difficult thing to do, because ourinternal kernel APIs change very, very rapidly.

In other operating systems, APIschange more slowly, since their development process is comparatively very slow.So, it makes sense, from a technical standpoint and just to save money, to getyour code into the kernel. It is maintained that way, and when changes happenwithin the kernel, your driver code will be fixed for you. If I have to changean API, I change it everywhere.

That approach also lets us seecommonality. If see that a driver’s doing the same thing as another driver, Imight merge them together in the shared core. That way, everybody benefits. Thenew driver gets smaller and easier to maintain.

That’s happened a lot. We’ve mergeda lot of drivers from a lot of different companies, and everybody’s happy aboutthat. The companies are happy, the users are happy with the smaller code base,and it’s easier to maintain.

The Linux Driver Project started outbecause of the perception that Linux doesn’t support many devices. It turns outthat Linux supports all devices out there. There’s really nothing manufacturedtoday that Linux doesn’t support, in a major consumer market. There are someone-offs, like some small video-capture devices I know of that we don’t havesupport for, but people are actually working on those.

The initial goal of the Linux DriverProject was to remove all barriers that could possibly be there, which weremostly managerial. To attain that goal, we said that we will write and maintainany driver for free for any company.

It turned out that not very manycompanies really needed that. A few niche markets needed it, and we’re writingsome drivers for them, but most companies had existing drivers floating aroundinside their company. Their challenge was typically that they needed to get itinto the kernel tree, and they didn’t know how.

So, the Linux Driver Project, overthe past couple years, has just been a big educational thing. I work with thecompanies and show them how to get the code in. I maintain the code, massage itand get it cleaned up, and then merge it into the kernel tree. That’s been themajority of the work we’ve been doing over the past two years.

We’ve also now codified a few waysthat we can play in the kernel. We can accept code into the kernel that doesn’tmeet our normal standards into a staging area. Everybody can clean it up there,and then the code graduates into the real stuff. That’s worked out really well.

Scott: That makes good sense, because someone coming in from theoutside isn’t necessarily going to know everything from coding standards to thebest practices for doing things the right way.

And so what you’re saying is thatyou’ve got a place to get that code in, where people who know the right way todo things can look for things that can be factored out of drivers, drivers thatcan be merged, and that kind of stuff.

It doesn’t force every hardwarevendor or everybody who needs to get a driver into the kernel to be a Linuxdriver development expert. Do I have that right?

Greg: Yeah, although we do have lots of documentation now. Wehave free books on how to write Linux drivers, and we have documentation ofthings like our coding style and how to submit code. If people don’t know whereit is, we’ll point them in the right direction. In case anyone’s curious, itall starts in a file called HOWTO in the kernel, so they can start there.

Even though we do have thoseeducation tools, though, we will also work with companies that want it. We havelots of people who have done this work before and who will do it for you.

It’s a great place for peoplewanting to get involved in the kernel to start out, because they can just run ascript to find some errors and go fix them. We have people that have startedoff just wanting to get involved who have ended up maintaining whole driversand subsystems over time.

Still, in the end, my goal isactually to work with engineers from those companies and have them maintain andown it, so they become full-fledged members of the kernel community. That’s theonly way we’re going to grow, and it’s working. The number of companiesinvolved in the kernel has grown year over year.

Companies want to get the most valueout of Linux, so I counsel them that they should drive the development of theirdriver and of Linux as a whole in the direction that they think makes the mostsense. If they rely on Linux and feel that Linux is going to be part of theirbusiness, I think they should become involved so they can help change it forthe better.

Scott: That makes sense. In a different area, I saw somewhere thataround half of the Linux code that’s being contributed nowadays is driver code.Do I have that right?

Greg: That’s the percentage. Like I said, everything’s in thekernel itself. We’re at something like six or seven million lines of code, andover 50 percent of those lines of code are drivers.

I think 30 percent isarchitecture-specific stuff, for things like processors and networking. Thecore kernel is like five percent of the overall code. Those numbers have stayedpretty much the same over the past four or five years.

We change something like 5,000 linesa day, which is just scary. Fifty percent of that change will be in thedrivers, and five percent will be in the core kernel. In other words, thekernel is being modified everywhere at that rate of change.

Scott: Do I understand correctly that the core kernel is memorymanagement, process scheduling, and those kinds of fundamental things that anOS has to do?

And the only reason we’re changingis because people want that change. It’s not like we’re sitting there andgoing, “Hey! Let’s rewrite the scheduler again!”

[laughter]

We’re doing this stuff because wehave to in order to survive, because people want it and because people need it.We do it for fun, but we don’t gratuitously change things. A big part of whatdrives that change is that what Linux is being used for is evolving. We’re theonly operating system in something like 85 percent of the world’s top 500supercomputers today, and we’re also in the number-one-selling phone for thepast year, by quantity.

It’s the same exact kernel, and thesame exact code base, which is pretty amazing. Nobody’s ever created somethinglike this before. And we’re doing it in a way that doesn’t follow thetraditional software design methodologies, which is fun.

I go and talk at a lot of colleges,and they’re changing their education system based on how the kernel has changeddevelopment models for large-scale systems.

Scott: Like you mentioned, the kernel runs on some of the smallestdevices, as well as some of the largest. What makes that work is a lot of stuffthat is or isn’t included, whether you’ve got fairly bare-metal, embeddedsystems, or whether you’ve got desktops and music players, or whether you’vegot systems that are optimized for number crunching.

The kernel is an interesting thing.When you talk to people who aren’t really in the Linux space, they often don’tmake a big distinction between the Linux kernel and a Linux distro. There’s ahuge distinction though, right? The kernel is a fairly small piece of a distro.

Greg: Sure, but consider Android, which threw away everythingfrom Linux except the kernel, and they built something totally new on top ofit. That’s a great proof point that the Linux kernel itself has to be really,really flexible to let people do something like that. It still meets the needsof a very big market, which is pretty funny to watch. From an engineeringstandpoint it was a pretty neat hack.

Scott: Give us some sense of the kinds of people who arecontributing to the Linux kernel. Obviously, there are hardware vendors. Thedistros out there are contributing–Red Hat and Novell, for instance, are bigcontributors. Give me a sense of the categories of contributors working on thekernel.

Greg: You’ve got a pretty good short list there. Some distroscontribute; some do not, and the same is true of hardware companies. Intel hascome on like crazy this past couple of years, and they’re now one of the majorcontributors to Linux. Consulting groups and educational institutions play abig role as well.

There are also a lot of companiesthat just modify code for one-off little things, making support better in theirhardware. Linux runs on yahts for a lot of automatic pilot steering things, andwe’ve had some contributions from there.

More than five or ten percent of theUS’s power is generated on turbines that are controlled by systems based onLinux, and those guys contribute some changes they need. It’s also interestingthat 20-25 percent of all of our contributions by quantity are done by peoplethat have no affiliation with anything.

Scott: The proverbial guy in his basement or dorm room orwhatever.

Greg: Yeah. They needed something fixed, so they did it and movedon. The quantity there is still large, but on the other hand, 75 to 80 percentis done by people who are getting paid to do it.

Scott: What do you feel like is driving change? You mentionedIntel, and Linux seems kind of tailor made for a chip company, because itprovides a great opportunity to put new things in a processor or a chip set.Contributing to Linux lets them build support for those kinds of things and tosee how they are behaving in a real production operating system.

I can kind of see that new hardwarearchitectures and capabilities drive a certain amount of change in the Linuxkernel. I can just see the creation of new peripherals driving a certain amountof change. What other types of change drive the Linux kernel forward?

Greg: Well, those two are major. New hardware accounts for themajority of our changes. We get more processors with more and more cores. Wehad to do a lot of work to make, 8-, 16-, and 32-way machines run really well,and now we’re running 8000 processor machines really well. The scalability isthere, and I actually know of people who have booted larger ones, but we’re notallowed to talk about it.

USB 3.0 was sponsored by Intel, andit was shown on Linux first–the first implementation of any operating system.Intel gets their hardware working well fast, and working with the Linuxcommunity lets them get it out to developers and other people who are makingdevices fast. The fact that they can do it for Linux much faster than they canfor other operating systems is a huge driving factor for what we need to do forthings like 10 Gigabit Ethernet or 40 Gigabit Ethernet.

Some of those speeds have requiredmajor changes to the way we do networking and the way we handle processes ordata flowing through the kernel. We also have to accommodate workload issues,like real-time workloads, which required big changes to the scheduler and theway we do locking and the way we do other things merged in.

Note that those are all requirementsdriven by people who are using Linux. The real-time guys want to use operatingsystems like Linux for their stuff. It also drives Wall Street and the ChicagoMercantile Exchange and the Tokyo Exchange and the London Stock Market.

Scott: What do you see on the horizon, in terms of hardware usecases that aren’t quite mainstream or available yet but that are going to drivesome additional kind of significant change all the way down to the level of thekernel?

Greg: 40 Gigabit Ethernet is going to need some changes, and thenetworking guys are actually working on that already. There are also going tobe more and more cores, better real time operation, and USB 3.0. The USB 3.0devices aren’t out yet, so once we start seeing more of those, we’ll startseeing optimizing things there. SSDs are great; they’re growing fast.

People seem to forget that changesthey want to the kernel that it seems like no one else will want very often endup being valuable to a lot of people. For example, maybe the embedded groupmakes some power management changes, and it turns out that the server guysreally want that, too. Then the server guys want better scalability, and itturns out that the embedded guys love those scalability changes, becausethey’re running dual-core ARM systems now.

Part of what’s helped us make thingswork is that it turns out that a lot of changes benefit everybody. We’ve becomesuch a flexible operating system by requiring changes to work for everything, andit’s made everything work even better.

There are lots of power managementchanges in the pipeline to be reworked that help address a lot of the issuesthe embedded guys have come up with that servers traditionally don’t have. Inservers, you can’t shut off that many discrete components, whereas in embedded,you can pretty much shut off every single discrete component in your system,depending how much you want to do with it.

Scott: I remember when we talked to the OLPC guys, one of theirrequirements was to be able to go to sleep, but not lose the networkconnection.

Greg: Mesh networking, right?

Scott: Yes. And they also needed super-aggressive powermanagement, because they wanted to get the most battery life possible, becausepower is such a big issue in developing countries. If I remember correctly,they were actually putting the processor in a sleep state between keystrokes.

Greg: That’s right; they were, but it turns out that the hardwareguys can do it better. If you look at more modern processors, the hardware isputting itself to sleep faster. In modern PCs and these new netbooks, the OSreally doesn’t have control of the power management stuff anymore, because thehardware can do it better.

It turns out it’s better to run asfast as possible to get your work done, then go to sleep, than to run your CPUat medium speed. The hardware guys realize this, and they know they can handletheir workloads better than the operating system can.

Scott: I don’t want to mischaracterize it, and I am intentionallyoverstating this to some extent, but when you talk to people at Intel, theyalways say, “Well, that should be in silicon.” It’s kind of like there’s thisdrive to put more and more and more in silicon, but at the same time, obviouslysoftware is growing at a geometric rate.

Talk a little bit about that balancebetween what ought to be in hardware and what ought to be in software, as wellas the notion that something that ought to be in one place today should maybebe in the other a year or two from now.

Greg: The hardware guys are always like, “Oh, I can solve that inhardware. Let me do that. You software guys aren’t ever going to get around todoing this.” And part of that is true.

If you look at the software adoptionrates of some operating systems, it’s very, very slow. So, the fact that theymake a change to the hardware, and they can save power today in the newprocessor, means that overall, the amount of power consumption in the worldgoes down.

People run old versions of Linux andother operating systems very frequently, so there’s sometimes an advantage tosolving a problem in both places. They can add it to the hardware, and thenthey also add it to the software.

In the kernel, we know we typicallycan’t change user space applications, so we have to make changes in order toaccommodate how they want to do things, even if we feel that they’ve done itthe wrong way. So, there’s also a tension there.

Some companies actually talk to theengineers really, really well. Intel and IBM are very good at this. They sitdown, and they talk to the kernel people once a year. “Hey, this is what we’rethinking of doing. How should we do this? How are we going to implement this?”And they get feedback from us directly. And that’s turned out to be a very,very valuable feedback loop.

It often works out very well interms of how chips are designed, and how the operating system will takeadvantage of that. Hopefully, we’re all talking to all the right people, andeverybody’s interacting well.

Scott: Let me spin back to something you said very early on, whichis that basically that all hardware is supported in the Linux kernel, eventhough there seems to be the perception that not all the hardware works.

Certainly, one of the things thatgets brought up is that if I buy a new device, it’ll come with a driver diskthat probably won’t have Linux drivers on it. Talk a little bit about where doyou feel like this disconnect comes from between the people working on thedrivers in the Linux kernel and those incorrect user perceptions, even frompeople who are fairly experienced with Linux.

Greg: That perception has existed for many, many years. Back in2006, I gave a big keynote showing how it was all wrong. Everything issupported, but, yes, that complaint comes up a lot. I joke with Jim Zemlin ofthe Linux Foundation about this issue all the time. If he publicly says, “We’restill working on drivers,” I always say, “No, we’re done.”

[laughter]

So, there’s a disconnect there. Partof the issue is that we support more devices than any other operating systemever has. That’s a fact that’s been verified by other companies. The problem isthat people only care about the devices that they have. Therefore, if yourdevice doesn’t work for some reason, you don’t care how many thousands of otherdevices out there work.

When I started the Linux driverproject, I’d been hearing this from all the major companies that were shippingLinux and cared about Linux. So, I went around to them individually and said,“OK, what do you need me to do? What needs to be worked on?”

Every single major hardware company,said, “Hey, you’re right. Everything works on Linux. We’re fine.” That’s shownby these companies that ship Linux on their machines. Dell, HP, and all the bighardware companies and laptop manufacturers now ship Linux, and we’re workingwith them.

Some of the disconnect is if you buya machine that doesn’t have Linux on it, getting it to work sometimes requiresweird BIOS tricks to get all the function keys working on a laptop, and thingslike that. New hardware or hardware when you’re updating a system should work.If not, then we messed up.

The graphics guys have issues thatthey’ve been addressing to help make that work much better. Intel hardwareworks wonderfully now on Linux. Their developers are very good. ATI and NVIDIAboth ship closed source drivers that work on all their devices. Those all workand have very good support systems. The drivers aren’t in the kernel, butthey’re out there.

Anything storage related ornetworking related has been working with Linux forever, and all new devicessupport Linux too. They have to, because that’s what those markets are goingfor.

Then you get to the tiny consumerdevices, which is where I like having fun, because I do USB. I work really hardto get everything supported, and we don’t know of anything these days thatisn’t. I was in Tokyo the other day for the Kernel Summit and walking aroundAkihabara and trying to find devices that we don’t support. We had all the kerneldevelopers there and we couldn’t find anything.

I know there’s a disconnect, but ifyou look on a mailing list for the Windows guys, they have a lot of driverproblems, where things aren’t working. Mac OS X does not support very manydevices at all.

If issues arise with exoticconfigurations, I’ll be glad to work through them all. If something doesn’twork, tell me so I know to fix it.

Scott: Is my perception correct that when hardware companies getinto Linux, they sometimes start out with a bare-bones driver? It seems thatoften, the piece of hardware will work, but it won’t necessarily handle thefull spectrum of services like power management with the device. Then they’llprovide more capabilities, gradually contributing code to use the device to itsfull capabilities.

Am I at all accurate in thatperception, or is it more often the case that the driver they contribute offersthe same level of functionality as on any other OS?

Greg: You’re right. Some things don’t work, although we’ve madeit easier for companies to write a driver that will do everything, because weput core functionality like power management into the core. Power management isnow handled by the core of the kernel, so you don’t have to add that. You justhave to add a few hooks in your driver and then you’re done.

The average driver for Linux isabout one third the size of an equivalent driver for another operating system,so you have less code to write and maintain. Still, to your point, there aresome new devices that come out whose value add is that they do X and Ydifferently.

Sometimes we don’t support that yet,because the manufacturer didn’t tell us how to do it or we have to figure outhow to fit it into the Linux framework to support it properly. Those littlewebcams are a very good example. New devices come out that can support featureslike additional bit rates, frame sizes, or screen sizes, and we need to go andadd support to the core of the kernel to support those.

So, your statement is very fair, andthat contributes to the disgruntlement you’ll see on my face at times.

[laughter]

If a brand new device that says itcan do X, Y, and Z is only doing X on Linux, we immediately start to try tofigure out how to get it to do the other things. Sometimes, we just have towork with the manufacturers to do that.

A lot of companies are gettingbetter at working with us to come out with drivers at the same time thehardware comes out. Sometimes we lag by six months just by virtue of that factthat Linux isn’t the first system they care about, so it takes us a while toget it going. Also, if you’re using an older distro, you may need to switchversions.

There are all sorts of configurationissues there, which is why Jim Zemlin of the Linux Foundation continues to saywe need to work on driver development, and I agree with him there. There’salways going to be work, so I think I have a lifetime job here.

Scott: Correct me if I’m wrong, but drivers never really leave thekernel, right? Is it more or less true that that’s one of the reasons thatLinux supports old hardware so well? That once it’s in the kernel, it’sprobably going to be in the kernel for a very long time, so I can grab asix-year-old laptop or six-year-old server, throw Linux at it, and have prettygood confidence it’s going to work?

Greg: Yes, and actually that’s turned out to be a problem.Because our kernel size is growing so much, we want to make changes, and we’rerunning into places where we have drivers for hardware that we know hasn’t beenmade in the past ten years. It can be complicated to excise those pieces.

That came up at the kernel summitlast week, and actually, I am going to be working on that. We’re going to movedrivers that we think are broken if we are pretty sure that nobody out thereuses them anymore. We’ll move those into the staging tree, keep them there forabout a year, and if nobody complains, then we’ll remove them.

On the other hand, if anybody showsup and says they need one of those drivers, we can use our source code tool torestore and test it. If there is at least one user of an old device in theworld, I will gladly maintain the support.

Scott: We’ve covered a lot of great stuff, and now I want to turnto a lazy interviewer trick and ask you what really interesting things Ihaven’t asked about. In other words, I’d like for you to supply both theinsightful question and the brilliant answer.

[laughter]

Greg: Well, just to touch back on that rate of change that Imentioned before, I just looked it up, and we add 11,000 lines, remove 5500lines, and modify 2200 lines every single day.

People ask whether we can you keepthat up, and I have to tell you that every single year, I say there’s no way wecan go any faster than this. And then we do. We keep growing, and I don’t seethat slowing down at all anywhere.

I mean, the giant server guys loveus, the embedded guys love us, and there are entire processor families thatonly run Linux, so they rely on us. The fact that we’re out there everywhere inthe world these days is actually pretty scary from an engineering standpoint.And even at that rate of change, we maintain a stable kernel.

It’s something that no one companycan keep up with. It would actually be impossible at this point to create anoperating system to compete against us. You can’t sustain that rate of changeon your own, which makes it interesting to consider what might come afterLinux.

For my part, I think the only thingthat’s going to come after Linux is Linux itself, because we keep changing somuch. I don’t see how companies can really compete with that.

Scott: Like you said, a lot of the rate of change in Linux as awhole is due to changes in the drivers. It’s fairly unique to Linux that itcarries the drivers in the kernel, so as the ecosystem grows, you’ve got moreand more people contributing code to Linux.

That’s not the case with otheroperating systems.

Greg: Right. And while that rate of change is consistent acrossthe whole thing, it’s also easier to write drivers. As I said before, yourdriver for Linux is one third the size of your driver for Windows, so even atthis rate of change, writing a driver for Linux is less work than it is forother operating systems.

In Linux, we’ve re-written our USBstack three or four times. Windows has done the same thing, but they had tokeep their old USB stack and a lot of their old codes in order to work forthose old drivers. So, their maintenance burden goes up over time while oursdoesn’t.

Also, as people change jobs, theygenerally stay with the Linux kernel community. I’ve been doing USB work forover 10 years now for Linux, and that’s a big body of knowledge. Within othercompanies, engineers usually move around, and that body of knowledge doesn’tnecessarily stick around.

For driver maintainers and driverauthors within Linux, it does stick around. The networking guys have been doingthat work for 15 years with Linux, and they know this stuff in and out. Theycan tell you what you need to change in your driver to make it work reallywell.