I used to do a lot of board level design, and do the operating system, the firmware, and the bringup, and device drivers, and all that sort of stuff. And it was sort-of mid-career that I started actually seriously programming these big machines.

+

+

Anybody here who was at the Monta Vista conference last year will find what I have to say today eerily familiar. And I'm sorry about that, but it bears repeating. I know my kids need to be told the same thing more than once before they'll do it.

+

+

[Slide 1 Overview]

+

So what I was going to talk about today - I'm going to assume that the audience here are people who don't actually work on Linux for Linux' sake, per se. You in fact use Linux as part of developing some other product. It's just a tool which you use to deliver something to your customers.

+

+

What I won't be addressing today is the ongoing day-to-day interaction which contributors such as yourselves should make with the kernel.org team.

4:00 - 5:00:

4:00 - 5:00:

+

+

That's the subject of another presentation, that I won't be covering that topic.

+

+

Certainly, we can get on with that during question and answer session. How long do we have in here, is it a full 60 minutes? ... 50 minutes. OK. There's probably about 35 or 40 minutes worth of material. I would encourage people to think of, and save up questions at the end. Generally I don't like these things to be a me-to-you. It's important that they're also you-to-me, because I am here to serve you people. You are my customers. You need to tell me how I'm going.

+

+

I'll do a bit of amateur economics about what I believe to be the economics which is unique to embedded development, and how and why that impacts the interaction that embedded developers have with the kernel.org team;

5:00 - 6:00:

5:00 - 6:00:

+

+

Discuss what we'd like to call "patch hoarding", sitting on live patches and why this can be a bad thing to do; Look at the reasons why you might be ... to merge your features into the upstream kernel rather than maintain them yourself; Look at which kernel version you might choose to use in your product development; Discuss what level of support the public kernel developers can give to you, and why they will support you in your product development; and then I'll spend a bit of time talking about how we, or more specifically I, make decisions about what things should be merged into the upstream kernel, and what the criteria are for making that decision.

+

+

[Slide 2 - The economics of Linux-for-embedded]

+

+

So looking at the economics of Linux on embedded systems,

+

I think the key difference with embedded is that usually there is no plan to upgrade the version of the kernel running on the product.

6:00 - 7:00:

6:00 - 7:00:

+

+

You develop the product, you build the software, the application stack, the kernel. And you bundle it all together, and mostly push it out. You may, maybe will update the kernel sometimes, but

+

you don't expect your customer to do it. It's an embedded product.

+

+

This is radically different from the desktop and server world where upgrades are expected. In fact you don't even know what kernel your customer will be running in the first place. He might be running a RedHat Enterprise kernel, or an OpenSuse kernel, or Mandriva or anything. And also the kernel will be upgraded across the lifetime of the hardware. That's a critical difference because then you need to plan for that upgrade.

+

+

Another difference between embedded and the server/desktop world is that the software for embedded is delivered directly from the hardware manufacturer.

7:00 - 8:00:

7:00 - 8:00:

+

It doesn't come from some other 3rd party.

+

+

Now one exceptional case in all of this thinking is the case of the hardware manufacturer. ... that's the Intel, ARM, Atmel, Analog, people like that, who make hardware which they then sell to other customers, and those customers will then deliver embedded products using that hardware.

+

+

Those companies are in a similar position to the desktop and server hardware manufacturers, in that they are motivated to make Linux run as well as possible on their particular hardware, so that they'll pick up more customers.

+

+

So what we're seeing at present in the kernel world is a lot of contributions coming from companies such as those, who sell hardware into embedded developers, rather than actually developing embedded products themselves.

8:00 - 9:00:

8:00 - 9:00:

+

So that is for similar logic as you would see for the desktop and server world.

+

+

Another thing we see - and I don't know why it is - but the embedded segment just doesn't seem to have as much money. They tend to be smaller companies, with shorter time to market, and quicker turn-around times. So they can't really afford the luxury of having people who are dedicated to working on the open source software. they have to be ..?? product out the door.

+

+

The net effect of all this is that, I believe, that the embedded segment is under-represented in the development of the kernel.org kernel, compared to the overall importance of embedded in the Linux world. It's still the case that most of the people that are paid to work on Linux are really keyed to work towards the server products.

Abstract

Andrew will summarize the kernel.org development and decision-making processes. Special focus will be placed upon how they impact the developers of Linux for embedded products, including the economics of using a modern kernel versus staying on a frozen older kernel version, and the economics of maintaining private patchsets versus merging work back into the public kernel. For those who choose to work with the kernel.org team, Andrew will look at how that can most effectively be done. Andrew works with Linus Torvalds and other members of the Linux development community (including open source developers and distribution vendors and industry contributors) to shepherd new features and quality improvements into the Linux kernel.

Biography

Andrew has worked at Nortel Networks' R&D labs, and Digeo Interactive, a maker of digital home entertainment products. He is currently employed by Google, working fulltime on the Linux kernel. His long experience with Linux development, and experience in the embedded realm, give Andrew a unique and valuable perspective on the issues facing embedded Liux developers.

Transcript

Thanks, Tim. Yeah I actually go back in embedded a long way. I
started my career as a hardware designer and I'm actually an
electrical engineer.

3:00 - 4:00:

I used to do a lot of board level design, and do the operating system, the firmware, and the bringup, and device drivers, and all that sort of stuff. And it was sort-of mid-career that I started actually seriously programming these big machines.

Anybody here who was at the Monta Vista conference last year will find what I have to say today eerily familiar. And I'm sorry about that, but it bears repeating. I know my kids need to be told the same thing more than once before they'll do it.

[Slide 1 Overview]
So what I was going to talk about today - I'm going to assume that the audience here are people who don't actually work on Linux for Linux' sake, per se. You in fact use Linux as part of developing some other product. It's just a tool which you use to deliver something to your customers.

What I won't be addressing today is the ongoing day-to-day interaction which contributors such as yourselves should make with the kernel.org team.

4:00 - 5:00:

That's the subject of another presentation, that I won't be covering that topic.

Certainly, we can get on with that during question and answer session. How long do we have in here, is it a full 60 minutes? ... 50 minutes. OK. There's probably about 35 or 40 minutes worth of material. I would encourage people to think of, and save up questions at the end. Generally I don't like these things to be a me-to-you. It's important that they're also you-to-me, because I am here to serve you people. You are my customers. You need to tell me how I'm going.

I'll do a bit of amateur economics about what I believe to be the economics which is unique to embedded development, and how and why that impacts the interaction that embedded developers have with the kernel.org team;

5:00 - 6:00:

Discuss what we'd like to call "patch hoarding", sitting on live patches and why this can be a bad thing to do; Look at the reasons why you might be ... to merge your features into the upstream kernel rather than maintain them yourself; Look at which kernel version you might choose to use in your product development; Discuss what level of support the public kernel developers can give to you, and why they will support you in your product development; and then I'll spend a bit of time talking about how we, or more specifically I, make decisions about what things should be merged into the upstream kernel, and what the criteria are for making that decision.

[Slide 2 - The economics of Linux-for-embedded]

So looking at the economics of Linux on embedded systems,
I think the key difference with embedded is that usually there is no plan to upgrade the version of the kernel running on the product.

6:00 - 7:00:

You develop the product, you build the software, the application stack, the kernel. And you bundle it all together, and mostly push it out. You may, maybe will update the kernel sometimes, but
you don't expect your customer to do it. It's an embedded product.

This is radically different from the desktop and server world where upgrades are expected. In fact you don't even know what kernel your customer will be running in the first place. He might be running a RedHat Enterprise kernel, or an OpenSuse kernel, or Mandriva or anything. And also the kernel will be upgraded across the lifetime of the hardware. That's a critical difference because then you need to plan for that upgrade.

Another difference between embedded and the server/desktop world is that the software for embedded is delivered directly from the hardware manufacturer.

7:00 - 8:00:
It doesn't come from some other 3rd party.

Now one exceptional case in all of this thinking is the case of the hardware manufacturer. ... that's the Intel, ARM, Atmel, Analog, people like that, who make hardware which they then sell to other customers, and those customers will then deliver embedded products using that hardware.

Those companies are in a similar position to the desktop and server hardware manufacturers, in that they are motivated to make Linux run as well as possible on their particular hardware, so that they'll pick up more customers.

So what we're seeing at present in the kernel world is a lot of contributions coming from companies such as those, who sell hardware into embedded developers, rather than actually developing embedded products themselves.

8:00 - 9:00:
So that is for similar logic as you would see for the desktop and server world.

Another thing we see - and I don't know why it is - but the embedded segment just doesn't seem to have as much money. They tend to be smaller companies, with shorter time to market, and quicker turn-around times. So they can't really afford the luxury of having people who are dedicated to working on the open source software. they have to be ..?? product out the door.

The net effect of all this is that, I believe, that the embedded segment is under-represented in the development of the kernel.org kernel, compared to the overall importance of embedded in the Linux world. It's still the case that most of the people that are paid to work on Linux are really keyed to work towards the server products.

9:00 - 10:00:

Things have improved.
10:00 - 11:00:

11:00 - 12:00:

12:00 - 13:00:

13:00 - 14:00:

14:00 - 15:00:

15:00 - 16:00:

Over time, it will help prioritize it was work.
You help people understand what things are most beneficial to the most users as they work on the Linux kernel.

I find that a lot of people are monitoring the mailing lists, websites, release announcements and that sort of things.
Often they come to conferences and people tell me things they didn't know.
And that's bad.
I shouldn't not know anything.
If you know something that is potentially of use to the kernel, to help or guide development, tell us.
And a way to tell us, is by sending an e-mail to you-know-which-mailinglist.
Tell us what you need.
There's no way we can do it if we don't know that there's a need.

Another way in which a person can contribute to kernel.org development is to review the patches as they go past.

16:00 - 17:00:

You can help us to check if the change will not damaging to embedded in any way.
You may see a feature which is close to what you need, but it needs a few tweaks to be more useful.
Please tell us!
Reviewing the patches, checking them for correctness and suitability for your application is direct
contribution and it's not a lot of work.

"Squeaky gates get the oil" is an old English phrase in that the person who makes the most
noise, is the one that gets the most attention. I'm afraid that's how kernel work is.
There's historically been a group of people who were very good at making a lot of fuss about
for example latency on a desktop and as a consequence, quite a lot of developers and
resources have been devoted to improving the latency on desktops for multimedia applications
and games.

17:00 - 18:00:

That wouldn't have happened if the first people hadn't appeared on the mailing
list and make us look bad.
So if you want something done; come on the mailing list and make us look bad.
Chances are, it will happen.

What a secretly fruitful group embedded is.
All ### and companies making embedded products.
They tend to sit on their own pile of patches
which they think are only relevant to your own product and you maintain them
out of tree. Consequently, these patches get larger and larger and your ###
becomes older and older and it becomes harder and harder to ###.
and basically never gets merged in the mainline tree.

18:00 - 19:00:

it's quite a bit of work to make a patch to merge with the mainline tree.
I hope the main reason why people sit on their work without sending it
upstream is because it's quite a lot of work to get it upstream.
But I've also heard of companies who want to keep the patches to themselves
because they believe they will be beneficial to their competitors.

sorry, that's not a good reason.
This is an open source product.
Yes, it will potentially be some small damage if software becomes available to your
competitors, but I think you have to see that as the price of admission to use
Linux. It's an open source product, everybody else is contributing into it, you
have at least the moral, if not a legal obligation to contribute to it
yourself.

Another problem of patch hoarding is that it gets increasingly expensive over
time as you have to roll forward through kernel versions.