The New Now Thing: Where Linux is taking the Embedded World

Out of nowhere Linux has suddenly became
the leading operating system for embedded
devices and applications. There was no battle. No CEO surrendered
his keyboard to an army of penguins. No stiff-suited guy talking to
a bank of microphones. No parades.

But it happened. Linux won the embedded space. The only other
OS vendor of any significance in that space, Wind River, still
makes a fine living doing what it has always done, which is
suppling semiconductor companies and various manufacturers with
embedded and real time operating systems (RTOSes), software
development tools and support, to the tune of a few hundred million
dollars in revenue per year. Not bad. Also not significant.

Go back a few years and two other leaders in the space were
Lynx and Ready Systems. Today Lynx is Lynuxworks, and Jim Ready -
perhaps the leading personality in the history
of embedded operating systems - runs MontaVista, one of the top new
Linux embedded OS companies.

Meanwhile Bryan Sparks, who made Caldera the first
well-branded Linux distribution back in the mid-Nineties, is
charging forward with Lineo, a Caldera spin-off that is on an
aggressive campaign to blow away not only its Linux competition,
but Wind River as well.

It's hard to find any hard numbers about Linux adoption in
the embedded world, just as it was hard to find hard numbers about
the Internet when nobody seemed to notice that it was taking over.
But it's happening. Just as surely, and just as rapidly.

The embedded processing market is enormous. It's everything
that processes bits and isn't general purpose computing. Look at it
another way: it's everything that could possibly connect to the
Ne,: the sprinker system in your lawn, the conveyor belt controller
at a distribution facility, all the medical instruments at your
doctor's office, the navigation systems in airplanet ... anything
and everything that relies on digital intelligence. This is the
world laying at Linux's feet. It's all Antactica now.

Why? The short answer is engineers. Hey,
they like Linux. It's familiar. It's small. It's net-native. It's
fast. It's reliable. And it's open to improvement by anybody who
wants to bang on code. What more could you want?

Lots, apparently. A faster, "harder" real-time Linux would be
good. Visual development tools. Ever-better ways for engineers to
support each other. The embedded world is open for renovation and
reinvention from top to bottom. Next to embedded, what's happening
in the rack & box world of client/server computing is
micropotatoes.

Where there's more than enough business to go around, there's
room for more than one leading market personality. Such is the case
in the embedded Linux world, where Lineo's Bryan Sparks and
MontaVista's Jim Ready are not only pushing development envelopes,
but also each other. While there is clearly much mutual respect,
there are also vast differences. At the risk of oversimplification,
MontaVista is all about development while Lineo is all about
business, which are exactly the two thrusts, literally, of Linux
development Both will surely succeed, but one suspects for largely
different reasons, which is why their two perspectives give us a
stereoscopic view of a marketplace that's as new as Now. Or maybe
newer.

On September 7 and 8 we spoke with both gentlemen at great
length, recorded the conversations and found ourselves with a
16,000-word transcript. What follows is about 4,000 of those words.
We e hope our edits do justice to their unique visions for this
whole new development universe.

My first job out of Berkeley back in '76 was with an early
mil-spec computer company: Rolm. Back then it was all about
cross-development. They would have development stations just like
we've had for embedded work ever since. What changed since then
were form factors. These Rolm boxes were huge.

Then we were fortunate in '81, when the 68k was announced. It
became clear that the minicomputer situation was going to change.
You would be able to build good computers out of micropricessors.
But there really wasn't a standard, off-the-shelf,
constant-across-microprocessor operating system.

Each was essentially a stovepipe that went up from the
silicon all the way through applications.

If it existed at all. So we thought, why not build an RTOS -
a real time operating system - that would, independent of processor
architectures, present the same interface to the customer whether
they were on an 8086 or a 68k? Then let them program in the
languages that were popular at the time - PLM or PASCAL - and let
customers pick and choose without changing their devlopment
environment. That was the original Hunter & Ready solution,
which is still valid to this day. Hunter & Ready later became
Ready Systems.

I had forgotten about Hunter.

Here's a totally weird coincidence: Colin Hunter became one
of the founders of Transmeta, where Linus now does his work.

It was an interesting fact to me that when I first met and
interviewed Linus, he brought the conversation around to embedded
devices. He did the same in his speeches. Wtthout giving away too
much about Transmeta, he kept saying that the most interesting
future for Linux wasn't in beige boxes.

Obviously he's a smart guy who can figure this stuff out. We
certainly agree that the most level playing field - the one that
favors Linux most - is the embedded space. Microsoft owns the
desktop. Servers are competitive: Linux does well there against
Solaris and NT. But in embedded, there's just Wind River. They're
the biggest player, but still a tiny fraction of the overall
market. That market is begging for standardization.

Wind River looks pretty big.

VXWorks is fine, but it's known only to its developers, it's
closed source and nobody knows how it works. So, as much as Wind
River would like it to be mainstream, it isn't, nor is it going to
be. Meanwhile there is this pent-up demand in the embedded area to
get normal. Linux is the shining light in this dark space. The
customers are like moths to a flame, in a good sense.

Let's get back to your history. How did you get into
Linux?

In 1992 I brought Linux into Ready Systems, coincidentally to
the building I am in right now. I brought all the engineers into
the cafeteria and said hey, I just downloaded this stuff that's
been produced on the Net by a bunch of guys, including this guy
from Finland, and I built these two floppies, doing raw writes out
to a Sun workstation. If you know anything about any of this, you
know that the probability of anything working was zero. But I stuck
it into a clunker 386 that I bought at a junk show and the goddam
thing worked. It came up. And it was UNIX. I was blown away. It was
a very sophisticated system. We got X Windows running, put it on
the network, and found acres and acres of functionality were
working, very stably. Relative to the embedded software of the
time, this thing was so far ahead in terms of functionality that it
was an amazing eye opener. We actually did an internal study about
Linux and wrote a paper called VNIX - I forget the spelling, but it
stood for "VRTX is not UNIX." It was basically about how to use
Linux in an embedded operating system.

What happened?

We had a company to run. We got busy and got merged and did a
bunch of other things. Still, it has been on
our minds ever since. Every month I got my copy of Linux
Journal. I was a very early subscriber.

We started in '94.

Yeah, one of my engineers showed me the very first edition,
and I immediately subscribed. I noticed that virtually every month
the magazine got thicker, the paper got better, the ads got more
sophisticated, the articles got better. You could tell that
something was going on. So my indirect measure of the goodness of
the Linux market in general was Linux Journal.
Seriously. It was my benchmark. You guys should know that you
reflected what was going on.

What happened to Ready Systems?

We merged with Microtec Research in '93. With Microtec we
were the largest combined embedded RTOS tools company - larger than
Wind. That was '93. In '94 we went public. Then in '95 Mentor
Graphics, the EDA company, in a future-looking move, bought
Microtec. And that was how the food chain worked.

Then what?

Without throwing rocks at Mentor, let's just say that the
business did not prosper for any number of reasons. But I kept my
subscription current in Linux Journal,
followed the Linux distributions. But the shocker was an article
about four or five years ago, by a guy from NASA Langley, about a
side-looking radar system. The previous version of this thing was
based on Multibus-I with two CPU boards, one of which ran VRTX - my
software. The other one ran some UNIX. He wrote that they managed
to replace those two systems with one running Linux. And I thought,
Hey! Something is going on here! VRTX was being replaced by Linux.
So, between that and knowing where the customers were going, we
could see Linux as a serious player in our space.

Let's jump to the start of MontaVista.

When I left Mentor Graphics I went to the guys who had
invested in Hunter & Ready in 1981, who by now were old friends
of mine, and said "I have not come to you with a new idea in
eighteen years." That was in March of '99. For the first six months
we were exactly what our competitors called us: seven guys in
Sunnyvale. We didn't hide that. We were testing everything like
crazy: customer requirements, hosts, targets, real time… We
had some preliminary hypotheses. We tested those and came down to
some interesting discoveries.

Such as?

We thought that real time would be a big factor, but there
were many others that were in fact more important in terms of
customer acceptance. Those needed to be addressed first, so we
re-ordered our priorities.

Starting with what?

The fundamental one was availability. If you go to Fry's and
get Linux from Red Hat or anybody else, it's a self-hosted X86 or
possibly PowerPC system that works fine for installing on a PC or a
Mac. But the classic paradigm for embedded systems is
cross-development, where the system is hosted in one place and
targetted somewhere else, because an embedded processor typically
is not capable of supplying its own development. So the very first
job was making Linux available and supporable as a cross-hosted
environment, which means your Solaris machine or Linux box would
let you compile, do a sysgen, build the kernel, and also compile
applications and then download them into the embedded
target.

How many hosts and targets do you support?

We support four processor architectures and three different
host machines.

I'm guessing the host, the development platforms,s are
PowerPC,X86 and Solaris.

Yes. But there are varieties on those as well. RedHat 6.1 and
6.2 on X86. Yellow Dog on a Macintosh. And Solaris - I forget the
revision number - on Sun. Then the targets are a subtle point too.
There's X86/IE32 family from Intel, and within that there are many
variants. The compilers are built for four or five subsets of X86 -
486, 586 and so on, that GCC - the Gnu compiler - supports. Those
are all uniquely built. So, when we say we support X86, it's a
broad statement. Over on PowerPC, there are even more variants, and
they are more fundamentally different than the X86 variants.

I know the 601 and the 604 are highly
different.

Yeah, and then you get into the Motorola and IBM embedded
controllers, which are great processors, that often don't have any
floating point. So it took considerable effort to put together
support for five or six different variants of PowerPC so that all
the libraries were built correctly. I should add that one of the
core values of Monta Vista is building all of this stuff out of
common source, and a common rev. We maintain a strict discipline of
building all of it every night.

If I look at all the permutations, you've got three hosts and
four targets-

-And if you include variants it's more like twelve to fifteen
targets.

That comes to 45 different directions that your souce can
compile.

That's correct. And if you ever get on the wrong side of that
you can see how easy it is to have drifting feature sets and all
kinds of conditionalities to account for.

How do you stay on the right side of that?

Software engineering. One of the things that distinguishes
MontaVista, if you look at the engineering team, starting with
Kevin Morgan, is that it comes out of the enterprise-level,
mainstream, state-of-the-art UNIX world. This is a fundamental and
deep skill in the company, and it is not one that is shared with
others in the Linux world. People who might be very strong in Linux
- God bless them - don't necessarily have this kind of
background in large-scale software engineering. That's an insight
we acquired by getting it wrong. No pride
here.

Are there people outside the company who have jumped in to
help with the project?

Indirectly, yes. For instance on PowerPC, we have done most
of the bring-ups, but not all of them. So the normal processes work
there. Which means we are big contributors to the PowerPC tree and
help maintain it. But we don't do it all. This is a community
effort.

You're positioned as pure open source.

Yes. And we are absolutely an open source advocate, so all
this work is always available on the Net immediately through FTP or
on CDs.

So if somebody wants to download a .tar ball of what you're
doing -

-They've got it. No tricks. No hold-backs, no passwords. It's
there. We have a moral obligation here. We've benefited greatly
from open source and ought to give back. But beyond that we think
it's a business advantage. We fully believe in it. Anything we can
do to make Linux - and especially embedded Linux - go farther,
faster, benefits us.

You seem to have a strong relationship at Motorola. Do you
have a cooperative development relationship with them?

They have been very helpful to us as a company, making sure
that reference hardware is available for us. When a new processor
comes out, FedEx appears with a board in hand. It's gracious on
their part, and we do our best to do the bringups immediately and
put them out on the Web.

There is a Motorola microprocessor in the Kerbango radio,
running your Linux.

Yes, it happens to be an 823. It's a strong architecture. The
IBM 405GP, for example, is also a very hot part, in the figurative
sense. Both are PowerPC. We have a bunch of customers designing
that part in. So we have a great relationship with IBM
Microelectronics, too. We did the Linux bringup on the 405 with
them. Right now we're the center of the universe for that.

Are there any embedded multiprocessing applications? I only
ask because I was involved with Groupe Bull in the mid-90s, when
they had finished their initial design work on the PowerPC, helping
design SMP into the chip.

Yes. Certainly in some of the server-type systems where the
distinctions between embedded and standard servers are blurred,
you'll see some SMP efforts. But it turns out that the same work
you need to do to make Linux SMP-compliant is the work that you
need to do to improve the hard real time preëmptability of
Linux.

So it can be done?

Oh yes. Our guys have done real-time UNIXes so many times
that they know exactly what to do. We jumped on the SMP work and
found that we could dramatically improve the response, by a factor
of thirty. That's just a huge increase in the responsiveness of
Linux in the real-time domain, based on enabling, on a single
processor, the SMP software, and using it to basically make Linux
more responsive. But it's done with standard Linux. That's the holy
grail. Since our whole positioning is 100% pure Linux, it's not
surprising that we're the guys who did this. Some of the other
Linux embedded companies, such as LynuxWorks, have another
operating system to sell you. Why would they put the effort in, if
it undermines their own OS?

What about Lineo?

Lineo acquired the guys who do RTAI, Zentropics, so they
think they've got a real-time solution there. But their motivation
to improve Linux is very low, because they have a company trying to
work a different way. Red Hat wants to sell you eCos for real time.
That's not Linux.

What's with eCos?

It's a kind of VRTX small kernel. They should have done
Linux, but they didn't, and now it's too late. They've invested all
their money into building eCos and have tried to reposition it as
some kind of Linux, but it isn't and everybody sees through
that.

This is part of what they picked up from Cygnus,
right?

Cygnus did it, and it was funded by Japanese customers who
wanted a small kernel. They did an okay job, but it's not Linux.
It's a rhetorical patch. A dead end. We're pure Linux, pure open
source. We spent the money, hired the guys, and went after making
Linux hard real time.

Tell me more about what it is, specifically, that you did to
Linux to make it more preëmptive and SMP friendly, and somehow
manage to speak to all these different instruction sets.

Good questions. Couple of things. When people think of real
time, they think of "well, how fast does it respond to interrupts?"
But the weakness in Linux, in terms of classic real time, is not
how quickly it responds to interrupts. It actually does a pretty
good job at that. The amount of time that the Linux kernel itself
keeps interrupts turned off is reasonably small. On a 300 MHz
Pentium it's probably 130 microseconds or something. It's not
supersmall, but it's very very respectable. So the issue of losing
interrupts is relatively trivial. The weakness in Linux actually
shows up after an interrupt occurs and you want to have a process
run because of that interrupt. The question is, how long should
that take? The weakness is that, in standard form, Linux is a
non-preemptable kernel. That means some low-priority process might
have just started a system call that will take Linux a long time to
finish because it does complex things. You're the pilot in the
airplane and you push the eject button, and Linux gets the eject
command and says "Oh, I'll queue that up, but first I'm going to go
finish this low-priority thing - flushing the memory or something."
Hundreds of milliseconds later, you're still waiting for action. So
you have relatively long response times for completing the circuit
from interrupt to actual process running.

What you want to do is make the operations in the Linux
kernel itself interruptable and preemptable,
so you can stop internal processing on that low-priority thing and
switch to the process level where you get punched out of the
airplane. It turns out that the process of breaking up the Linux
kernel - so it can leave where it was and come back later -
requires identifying sections inside the code where this is
possible. This is the same thing that makes running multiple
processors possible. It turns out that the same engineering
investigation and thought process we use for SMP can be utilized to
make the Linux kernel preemptable and very much more responsive. So
we're not hacking out the middle of the kernel on this. We're just
enabling, on a single processor, the SMP facilities, but not for
SMP. And it works like a charm.

How does the rest of the community take that?

We're being very careful here. First, part of the charm of
Linux is that you can do whatever you want with it. However, we're
being very humble. This is a prototype. As we announced, our source
will be on our Web site, which is our tradition. And we are
offering this as a starting point for a very good set of
capabilities that we are suggesting, in the most humble way, that
2.5 should consider. Linus always says "show me the code." Well,
we're obeying that rule. We think this is a significant improvement
in Linux in general, and showing the community the code, for all to
see and play with, is the right and only thing to do.

So you're submitting it for peer review.

Absolutely. Out it goes, in the best of the open source
tradition.

How do you make your money?

Service. We get technically selected, just like any embedded
OS company would be. We sell, on a per-seat basis, subscriptions to
Hard Hat Linux, for every engineer on a project, for one year. Like
a magazine. This is how the RTOS business has always worked, except
for where you assign the perceived value. The customer wants a
functional RTOS and the company behind it. Whether the code is free
isn't relevant. As an issue, it's naive. Our prices are industry
standard - about the same as Wind or anybody else.

So what you're recognizing is that what you were always
selling in your earlier companies, was not the binaries, but the
relationships, the expertise, the guy on the phone.

Exactly. And the road map. And everything else a customer
wants to know is there.

What about royalties?

We think royalties are a very uninspired way of doing
business with an engineering team. We just don't do it. By the way,
in the business nobody paid royalties anyway. They always did
buy-outs or other deals that negotiated royalties away.

So you want to de-complicate life for the engineers.

Engineers like stability. So the subscription model
de-couples the rev of your product, which you will do from time to
time in any case, from your revenue stream. How many revs happen in
two years isn't the issue. Known costs and current software are the
issues. Engineers want constancy and good supplier relationships.
For classic embedded, with engineering involved, this is a superior
way to do business.

You have fewer interrupts in the conversational process
between vendor and customer.

Absolutely. Our larger customers beat this into us. In
effect, they threw our catalog in the waste basket and said "We
want to do business with you, but with a structured,
engineer-to-engineer relationship, based on your products, but much
deeper than that. We want better visibility, better responsiveness,
influence on your roadmap, and a closer relationship all around."
We got more money from those customers rather than less.

Your customers then, are engineers.

We have a B2E or E2E way of working. We sell to engineering
teams. That's who we automate and outfit.

I have a friend who likes to substitute the letter
W for the number 2 in those X to X business model acronyms. They
think the problem with "to" is that it presupposes a distribution
rather than a cooperation relationship. We don't do business "to"
people. We do it withthem.

That actually is a lot better. I like EWE: engineer with
engineer. I could buy that. It's a multifaceted business
relationship, based on technology and product. Our customers get
technology seamlessly from us, in the best possible form, which is
EWE. We answer the phone. They know where we're going. We know
where they're going.

What this also says to me is that Linux is a living operating
system - a network of "W" relationships. It's organic, and
companies like yours are organs in the growing body of expertise
about Linux.

Yes. You have to lead and participate and follow along all at
once. Given the way that Linux has evolved, without central
control, is so remarkable that you've got to believe that there are
some forces at play that are working so well that it would be very
rash to think the process could be helped along or improved. It's
fascinating to be part of, and it's certainly something I would not
want to bet against or be on the wrong side of, honestly.

One of the things I've observed about the Net is that it was
made by this same phenomenon. It is embodied by its creators with
three virtues: nobody owns it, everybody can use it, and anybody
can improve it. What you're describing here is more of that same
thing, and one way to make a living in it. It's in Lineo's interest
to take apart your real time Linux release and put it through that
same peer review process.

Yep. You're absolutely right about the Internet, and about
Linux. This thing is spreading like wildfire, and its in our
interest - along with everybody else's - for us to pour gasoline on
it. These things have very explosive properties. And we have to
contribute to the ongoing explosion. Michael Tiemann at Red Hat
tells the truth when he says there is a proven model here and it's
the scientific method. You do your experiments, show your results
and so on.

Eric Raymnond talks about that too.

Yes. In this unruly and wild Linux community there is
something that propogates very quickly through companies like
MontaVista that make it possible from a business standpoint to
depend upon it. It's a force. We have to be
careful not to act as a filter in front of all of this, but rather
to act as an enabler, so people can tap in to
this phenomenon in a sane way and still get their job done. That's
our purpose in life. We want to enable the engineers to do what
they want to do anyway, which is use Linux. Boy, that's a deep
phenomenon.

Are you saying Linux is ubiquitous with your
customers?

We go into these places and virtually everybody we talk has
fooled with Linux at home, at the very least. All we have to do is
give them the justification to use Linux at work and you've got all
these guys working on your behalf.

One of the things I observed about the IBM announcement, when
they essentially declared themselves a Linux company, was that this
was clearly a company that was now in full compliance with its own
engineers.

Oh, there is no question that this is a phenomenon of
software engineers. Look at Sun's success in the early years. There
is no question that they won the hearts and minds of software
engineers. If you were an engineer sitting there at a VT100 with
VMS at the other end of it, and the guy down the hall had a Sun
workstation with a big monitor and a mouse and multiple windows
open and running UNIX, you will do everything in your power to get
one of those things.

I like your way of characterizing Linux as a fire you pour
gasoline on. For years I liked to talk about the only kind of truly
effective marketinn. The logic went like this: A) Markets are
conversations; B) Conversations are fire; and therefore C)
Marketing is arson. What you're saying is that if you're not
committing arson, you're not really committed.

That's not a bad analogy at all. With good old VRTX, we were
trying to practice marketing arson, trying to set fire to the
world, and we did okay. But they were just our efforts alone. If we
scaled it up a bit, we had the RTOS industry, alone, trying to set
that fire. It's a respectable business. In fairness, Wind is a $300
million company through acquisition and everything. But it has
taken forever to get there.

Now, however, we're part of a larger phenomenon called Linux,
which is many, many conversations going on all at once. It is a
wildfire. That's why it's so fundamentally different. It's
instantaneously worldwide. It's very deep. It occupies the minds of
countless engineers, naturally selecting the best and the
brightest. It is all of these things. It's a whole lot of fun, I
can tell you that.

Will you do acquisitions?

Sure. That's a real possibility. But I have been through two
acquisitions, and I can tell you that in the best of times they are
very hard to do. You get what you get, which includes good guys and
bad guys, old agendas, different code bases. Lineo's acquisition of
six companies has to be extremely difficult. We've grown to 100
people organically. And it shows in results. We're producing
software.

Are you making money?

No. We're a start-up, and we're just trying to grow. This is
a land grab. We have customers. We are doing well. But clearly when
you establish European and Japanese offices, you've got a lot to
eat while you ramp up. Investors don't put the money out to have it
sit in the bank. They want it spent.

What does your plan show?

A turn to profitability at a future point. And we are
exceeding our plan.

How many customers do you have?

Fifty.

So you have EWE going with fifty customers.

They aren't contracts. They're orders. Customers issue
purchase orders. They buy subscriptions. We do consult, by which I
mean that customers buy NRE (non recurring engineering). But if you
were to think about embedded systems, you'd think hmm... telephony
equipment, internet infrastructure equipment, etc. That's who we
have. Companies who make gizmos.

Do you think one industry will turn out to be a more
significant embedded Linux category than the rest?

We see three categories. One is what we call A-to-Z embedded.
That's embedded controllers and instrumentation and gadgets of all
kinds. But the driver of microprocessor technology is what we call
"either end of the Internet." The infrastructure equipment guys -
the Ciscos and Alcatels and Nortels and fiber optic people
- those are a very strong component of our business. Then the
third cagegory is things like the Kerbango radio: things that
attach. Appliances and all that. God bless Linux, it looks real
good all over the place. You'd think, given the diversity here,
that one kernel would have a hard time stretching over all these
different applications and situations, but Linux is doing it.
There's a communications revolution going on. The Internet is part
of that and Linux is part of that, and both are driving
requirements that are falling out in to general purpose embedded.
It's very interesting: the two earliest customers were one
infrastructure company and one appliance company: Kerbango. Both
drove us to make Linux better in the same fundamental ways.

Two ways. The first is that we started Caldera back in '94.
Back then we bought some business from Novell and started selling
DR-DOS. We intended to use it as a thin client for cash registers
and applications like that.

You own the original DR-DOS, then.

Caldera bought that in '96. We had a lot of interest in
embedded, and DOS looked interesting to us for that. There was a
need in single-board computers and non-PC-like devices.We ended up
finding kind of a business there, and that's where I ended up
betting my career. We spun off two companies. Caldera systems,
which you know, then Lineo, then we collapsed Caldera, so there are
just the two companies. It was confusing. For a while Lineo went by
the awkward name of Caldera Thin Clients. We did some DOS and Linux
stuff, but got more and more interest in Linux. Clearly there was
something going on here. So we said "let's turn the ship here."
Which we did. You won't see anything about DR-DOS on our Web site.
We just don't do it anymore. Our whole business is embedded
Linux.

What was the time frame here?

This was around late '98. That's when we said this is it, and
picked a new name and started moving forward seriously.

Good timing.

A lot of it was luck, and being in the right place at the
right time, and having revenues from the DOS business to fuel the
early stages of our embedded Linux product.

You were the first out of the gate with Linux distributions
too, in a way.

We were.

And you got kinda snookered by Red Hat. I take it you're not
going to let that happen again.

It's not gonna happen here. I learned a tremendous amount. It
was very humbling. We were the leader. I made a lot of mistakes. We
got passed in the last lap and somebody else took the checkered
flag. Pick your metaphor. Boy, that's not going to happen
here.

What exactly did you learn?

One was that we did not have a culture that pointed at a
competitor. We were leading in the space and not looking out for
our back, where everybody else's guns were pointed.

You had the pioneer's problem.

There are two strategies, of course. One says that first
movers always win. The other says second movers win because they
take advantage of the first mover's mistakes. The second is what
happened here.

Happened with DOS, too. DR-DOS was first, but
MS-DOS won that race.

Here I feel like this is my second chance. And it's not just
me. There are a lot of second-generation Linux people here. People
from Linuxcare, even Red Hat. They've been there, done it the first
time, and made mistakes we can all learn from.

And you're applying what you learned.

Absolutely. I feel that we are executing about as well as any
company can. I am so proud of the people we have and the job that
they're doing. Ultimately that's what wins. You can have a great
pitch - our pitch is okay - but you have to execute. You need
top-line growth every quarter. You need good customers who come
back because you do what you say you're going to do. We have that.
It's real simple stuff.

I'd like to look at the embedded space we've had for twenty
or so years, and how Linux is changing that.

It's amazing. We're growing and assimilating employees as
fast as we can, yet the demand placed on us by semiconductor
manufacturers, and their customers, still overwhelms us. So that
tells me that, well, maybe Linux hasn't won, but it's certainly in
extreme demand.

So it's still early.

Yes, and you have to look at Wind River. If you saw their
last quarter's revenue... holy flippin' mackerel. They made a ton
of money.

How many people do you have now?

Two hundred sixty. About half are here in Utah. The rest are
outside.

A bunch would be in your
acquisitions.

We did six acquisitions this year. And we have grown staff at
most of those acquisitions as well. And we don't run them as
separate companies. We run them as part of our company. But the
offices represented by those aquisitions have grown. We bought
technology, some of which was in embryo. We had to kind of beef up
that technology, but in every case the opportunity is just
huge.

And you've attracted a lot of
attention.

Oh yes. You'd expect me to be optimistic, but I think: gosh,
if one tenth of what comes to our door pans out, we're in great
shape. Linux is going to make huge inroads in this space.

Tell me more about the demand from the semiconductor guys.
Does it sort out to their customers categories? Or is it something
that comes out of your working relationships? Or both?

We are extremely close to our customers. Our strategy from
the start has been to establish the strongest possible
relationships with semiconductor manufacturers, so we get into
their roadmap, and target what would be most interesting for
their customers to run Linux on.

How do you adapt to the verticals?

We verticalize some of our solutions. We're already doing
that internally. But the main idea is to provide ubiquity on
embedded platforms.

Which platforms in particular?

SH 3,4, StrongARM, ARM in several variations, almost all the
Motorola chips - MCORE, ColdFire, Dragonball, PowerPC… Of
course the X86 kind of comes for free. That comes to ten, I
believe.

So your horizontal product is Embedix.

Embedix provides a whole list of services. Not only the Linux
stuff, but hard real-time. Jim Ready and others do soft real time,
but we do hard real time. Tymsys does too; but we do hard real time
with extensions. We also do soft real time if people want that. We
have a browser that we're converting into a more generic graphical
user interface with cascading style sheets, HTML, Javascript and a
whole bunch of other stuff that sits on top. This all works
horizontally. So as we start verticalizing some of our solutions
we'll get stronger offerings in those areas. We already have design
wins in routers. DSL, VPN, IP security stuff, plus verticals around
some of the devices we're working on.

You say you work closely with your customers, which are
primarily the semiconductor companies. Tell me more about how that
works.

We are literally part of their team. Their wins are our wins.
Their customers are our customers. We want direct relaitionships
with their customers, the OEM device manufacturers.

So if Motorola sells to GM or Ford, you want to be in there
with them, so those companies are your customers too.

Yes. And this is the relationship we have with all the
semiconductor manufacturers.

Who's showing the most interest in Linux?

It's across the board. But there are some surprises. If you
asked me a year ago, I would have told you the sweet spot for Linux
was Internet appliances, routers, gateways...

Stuff close to the Net.

Right. Stuff that takes advantage of the core competencies of
the Linux communities. But the areas that have been surprising for
us - and we're pretty much the only ones who do this - have been in
consumer electronics. I'm talking about Dragonball, MCORE ColdFire,
ARM and those devices, which run in digital cameras, cell phones,
handheld devices and other consumables. We are working with a lot
of companies in those areas. We can really run Linux on those
devices. You don't have an MMU? You constrained by 200k of RAM? We
can fit in that. The interest is extremely high in that space. When
you consider that our competitors in that space - CE and Palm
- lead with a GUI, we're looking good with Linux, which
doesn't have an inherent GUI. So we manufacture one with a browser,
Java, whatever.

But the demand is for Linux in this segment.

Exactly. It starts with customers wanting to run Linux down
at this tiny scale. It's fascinating.

What about process control, industrial automation and stuff
like that?

It's a market, but it's not growing, and it doesn't turn over
very fast. For those spaces we offer VXworks and Psoft migration
toolkits, to make the switching costs easy. Our hard real time
plays real well there, too. In fact, we're doing big contracts with
the government on that kind of thing. We're doing a big contract
with NASA for flight simulators, weather monitoring for the FAA...
things like that, through one of our acquisitions. But these aren't
the surprises. The big surprises are at the bottom of the pyramid
where the parts are small, the demand is high and the volumes are
enormous. Things you see everywhere. So obviously those are big for
us, and I think they're big for Linux too.

What's your key offering there?

An interesting road map. When you buy into Linux, you buy
into a fast moving technological roadmap. We augment that with
technologies we either acquire or build to make more interesting to
those verticals. It's a fascinating thing. Previously they were
either buying from Palm or CE, which are proprietary RTOSes with no
road map at all.

What is your position among your competitors, in respect to
customers.

I can't imagine anybody talking to more customers and
partners than we are, just because we're large. We get a lot of
feedback on what customers are asking for. Right now we're busy
just backfilling customer requests on a road map. We have to get
ahead of that. We need to lead in embedded Linux, from the
customer's perspective. Right now we're just satisfying customer
requests. That kind of drives our road map. But eventually we'll
have to do more R as well as D. I think we'll get to the R in six
months or so when we get ahead of the curve a bit.

I would imagine that the D kind of pulls the R along.

The customers are telling us "we need this," and we have a
choice to make. Is it on our road map? No? What do we do? Build?
Buy? License?

What's an example of that?

Bluetooth - the hundred-meter wireless Ethernet speed system.
The question was, do we build a bluetooth stack? Because we were
thinking about building one. Then out of nowhere IBM announced an
open source Bluetooth stack. Thank you very much.

How do you contrast what you're doing with what MontaVista is
doing?

First, we're not a pure open source play. We license third
party products with IP - intellectual property. We have our own IP.
We don't open source our browser. It's really good, and we don't
see any reason to open source it. The benefits don't outweigh the
liabilities. But I guess the main differentiation is that we lead
with product and backfill with service. We are highly motivated to
build a Wind River-like tool chain.

What you're about, really, is tools.

You will see an ever-increasing series of tool chain
enhancements coming out of Lineo. A bigger and bigger SDK.

So your orientation to open source is to cherry-pick those
products and tools that are useful to you and building a suite of
products, primarily tools, for shaping Linux into something useful
for your customers.

Those tools we build. But what I'm talking about are design
automation tools. Graphic configurators. Things that run on the
device. Embedded apps. Our question constantly is, "How can we make
the time-to market for our customers as short as possible?" To the
degree that we're successful there, we will steal business away
from Wind River.

It sounds to me like you're building a Cisco.

We're not doing hardware.

I was thinking of the kinds of customers you're working with,
your approach to acquisitions, your determination to lead the
market no matter what, your customer focus. I'm amazed, for
example, at Cisco's ability to acquire companies and not have
trouble swallowing them.

Right. You can imagine that we had a strategy for acquiring
companies while we were private. We wanted to learn how to
integrate. And you know what? Buying companies is hard. Signing the
document is easy. Integrating personalities, cultures, ownership
responsibilities, technologies, all those grungy things, is hard.
We learned a lot. We didn't lose anybody in the process, but it was
a lot harder than I thought. It took us two months more than I
thought it would take.

It seems to me you went about your acquisition strategy based
on target device size scale, from small to large.

That's exactly right. We said "We want to do
that. How do we get
there?" We had holes, just working with a
generic embedded OS. We didn't have anybody at Lineo with real time
experience. We needed to go acquire that expertise.

How do you integrate Linux and your hard real-time
capability. Do you have a name for that?

It's folded into our Embedix product, and it's a feature of
our tool chain. You want real time on this device? Here's how you
select and configure it.

So when an engineer goes about designing for a particular
processor, application and board, and real time is one of the
things he wants, he implements it through Embedix tools. But it's
not Linux, necessarily.

Yes. But that, by the way, we open source. That whole thing
is available to everybody. We followed the RTAI - real time
application interface - standard. We open sourced all of that.
You'll be seeing more from us there. Actually we were ahead in real
time for a while, but let others move in and define the
market.

Who?

Jim (Ready). He talks about it all the time. And he doesn't
even do hard real time.

They announced their preëmptive kernel yesterday, which
they're vetting with the Linux development commmunity.

Yeah, he's kind of defining that space, but we're going to
fix that. That's why we acquired Xentropics, because they're the
ones who did our hard real time.

Where do you see Linux moving the embedded space?

I think by offering Linux on embedded devices we raise the
bar on functionality that customers will now require. There is this
dichotomy in the world where a lot of companies
say they are customer driven, but in fact
customers only ask for what they are aware of. They don't know
their necessities until they become aware of what's really
available. The context changes. Adding Linux to Internet devices of
all types will seriously raise the bar in functionality that people
will demand. An example. In our first release of Embedix early this
year, we matched VXworks in functionality. In our fist
release.

I'd like to go back to the power of invention. The real
marketing principle isn't "necessity is the mother of invention,"
but "invention is the mother of necessity."

That's right. We want to show customers what they're missing
out on. Linux does that on a huge scale.

It also seems to me that Linux has the same network effect
that the Net has had, only starting a bit later. It's kind of the
other shoe dropping. Same phenomenon, just a later stage.

Our mission statement is, "we unleash the imagination of our
customers to innovate for their customers." That means we want our
customers to innnovate in ways they didn't even think they could
before.

So your inventions lower the thresholds of their
inventions.

Yes. There's another thing. We are going to provide a
flexibility that these guys have never seen before. Before Linux
came along, and a customer needed a feature, they went to Wind
River and eighteen months later they might get it. The whole thing
is way too stymied. Between Linux and our SDK we're really going to
unleash the imagination of our customers. And they will only demand
more and more features, which we will be in the best position to
provide. We want their time to market
requirements set the pace. Not ours.

You mentioned doing this surprising work with small
consumable devices. Are you working with some of the Asian consumer
electronics guys?

Nearly all of them. We have fifty people in Asia. A large
presence in Japan, plus another fifteen or so in Taiwan. Most of
the investments we did in our round of financing came from Asain
manufacturers. Acer, Samsung, Mitsubishi, a bunch of others. And we
are working with a bunch of people. I go to Asia a lot. Fortunately
we're kind of all alone over there. It's hard to get in. We got in
partly by acquiring a company in Japan that was doing embedded
custom engineering. Then we organically grew a Taiwanese office
with native personnel.

I'm interested in scale. A huge company like Hitachi might
have a huge relationship with a customer like Sega. How is there
room for guys like you in there? I guess there was room for Wind
River, though.

I don't think our business model is all that different from
Wind River's. They, being such a dominant supplier, were able to
extract more money from the semiconductor manufacturers than we
can; but that's part of our strategy - to commoditize the space a
little bit. When they talk to investors, they talk about raising
per-unit value. When I talk to investors, I say we're here to
commoditize system software for interenet devices. The model is the
same, but we're really putting price pressure on them. We not only
give them real competition, but make it much harder by coming in at
a lower price.

You still have enough room in there to make money on tools
and services.

We do request a small royalty, adding our own third party
intellectural property on the device. But again our strategy is
customer acquisition through commoditization of the space, sell
tools, and grow. But in terms of what we sell, we're similar to
Wind River. It's just that a smaller percentage comes from
royalties.

You charge on a per-workstation basis for the SDK.

Yes, and we have royalty-bearing components in there.

You pass those costs through.

That's right. And that's very standard for the industry. We
may do more licensing. We may do more acquisitions in the area so
we can keep the price low but still provide the functionality so we
don't have to carry the royalties back.

How big do you expect this category to get?

There are a bunch of analysts who say big, big, big. We want
to grow faster than the market as well. Just keeping pace with the
growth of the pie, we're at a rate of 60%

Do you see The Smart Home finally coming, because of Linux's
standardization properties? Or will the firewalls of ignorance that
surrounds every specialty going to still prevent that from
happening?

I don't know. I brought a bunch of CAT-5 cable to the drop
point at my house and the phone guy made such a hairball of it that
I had to redo it myself. But some of the standards are helping a
bit. HPNA, a home networking protocol that runs over twisted pair,
helps. Bluetooth helps. We're working with a company we're hoping
to close that is doing stuff with home gateways like I never
imagined. We have a TV set top solution that we've sold quite a
bit. But this is something else. We're talking about something
wired to appliances that does far more than turn them on and off.
It does service and warranty control. Your AC unit can say its
compressor is going and to send for a guy to avoid the insurance
hit. That kind of stuff. I'm crossing my fingers that we get that
one closed because it's just so cool.

That's part of the promise of XML. Devices can talk to
devices, live, in an XML stream.

Can you imagine this little control box calling a technician
without the customer knowing what's happening until the technician
says "I'm here to fix your fridge." This isn't ten years out. It's
next year. We giggled a couple years ago that somebody in Japan put
a browser on a fridge. Now, so what? We can put one on the
microwave. There's your recipe. Your documentation. There was a
piece out of Xerox PARC a long time ago that was right on. We are
living that now. And Linux will only help there. Great internet
infrastructure. Remote diagnostics. Network management. VPNs built
in. If you're a telecommuter, Linux is great. We actually sell a
little hardware box that's a VPN router. We sell it like candy.
It's unbelievable.

You sell a lot of set top boxes?

Yeah. You can browse the Web on these things, watch enhanced
TV, get your email, or whatever. We sell a boatload of that.

What else do you open source besides your real time
component?

Go to www.opensource.lineo.com. There's quite a bit there.
Utilities that run on an embedded device, like BusyBox. A bunch of
drivers that we think would be interesting to others. We have a
pile of open source projects.

Are your fingerprints going back on the Linux kernel?

Oh yeah. We give a lot of patches back. We want them to take
more. But sometimes the patches we have, to fit on very small
devices, aren't very mainstream.

What has Linus done for the embedded Linux space, by talking
up mobile and other forms of development, and working with
Transmeta?

I think he's done a tremendous amount. He's helped Transmeta
a great deal too, giving them branding and recognition. And he
really seems to enjoy it there.

Last question: What kinds of companies will survive and what
kinds will not?

I think there will be tremendous revenue opportunities in
this space. I cannot imagine that we'll be the only ones succeeding
here. Lots of new companies seem to be rushing in, but it will be
tough for a lot of those. We think we have found an interesting
model that investors and investment bankers like, and we'll see if
we can keep executing on it. But we won't be alone. And don't count
out the larger companies - Red Hat and others - will make some
moves.

Any last thing you'd want to add?

You'll like this one. The cluetrain stopped at Lineo, and by
God, we got on.

Doc Searls
(doc@ssc.com) is senior
editor of Linux Journal and coauthor
of The Cluetrain Manifesto.