Embedded Systems, Linux, and the Future

Predicting the future is a tricky endeavor, and an often-failed one, at that. So
I won't try to predict the future in this article. Instead, I will present how the
traditional players of the embedded systems field are currently positioned,
what impact Linux is having, and what forces will decide where the field is
going. I will also cover the various initiatives, moves, and trends trying to
influence Linux's move into the embedded systems field, and what role the open
source and free software community (along with embedded system
developers) should play to ensure that Linux is the best choice for an embedded OS.

The Traditional Embedded OS Market

Before anything useful is said about this market, one has to keep in mind that
for a long time, 50 percent of embedded systems were running custom-made, in-house
operating systems. That's an important figure, as many of the engineers deploying a
"roll your own" OS are increasingly attracted to Linux. So beyond grabbing
market share from established embedded OS vendors, Linux is also penetrating
the "last frontier" of the embedded OS world.

I don't intend to discuss the history of the embedded OS market in detail.
Rather, I'm more interested in analyzing how this market has reacted to the
surge of enthusiasm generated by Linux's increased use in embedded system
development.

Contrary to the workstation and server market, there has been no traditionally
dominant embedded OS vendor. Though WindRiver Systems (WRS) is considered
the most important embedded OS provider (and despite Microsoft's attempts to enter the
market with Windows CE), the embedded OS market remains fragmented, with no clear leading
company. Nevertheless, a few companies, including WRS, QNX, and Lynx,
have occupied an almost equal share of this market.

Generally, the traditional embedded vendors and publications have been
dismissive of Linux. Jack Ganssle, a widely respected figure
in the embedded systems arena, recently wrote in his monthly column
that "Despite all the
hoopla about Linux, I think the future is far too hazy to predict its
dominance in the embedded world." The future may be hazy, but established
embedded OS vendors haven't been taking any chances either.

The first to move was Lynx Real-Time Systems, which announced in November 1999
that it would be releasing a new distribution, called "BlueCat Linux," aimed at
embedded applications. Later, in April 2000, in what was a stunning move, the
company rechristened itself into "LynuxWorks" to reflect its newfound
allegiance. In January 2001 at LinuxWorld New York, the company
demonstrated a capability it added to its proprietary Real-Time Operating System (RTOS), LynxOS, to run
unmodified Linux binaries. LynuxWorks was thereby promoting a two-fold solution,
where customers wanting hard-real-time performance should use LynxOS, and
those wanting a non-real-time, royalty-free embedded OS should use BlueCat.
The company's strategy is unchanged to this day, and LynuxWorks continues to
provide both LynxOS and BlueCat.

QNX, too, has seen the skies change, and made some moves of its own. In April 2000,
around the same time LynuxWorks found its new name, QNX announced that it would
make its OS available free for download for noncommercial use. True to its
word, the software was made available to the public in September 2000 and
continues to be available here. Contrary to
LynuxWorks, however, QNX decided not to touch Linux at all. Instead, it has
been preaching that Linux is not adapted to real-time applications. In a
whitepaper entitled "Real Time or Real Linux?," for example,
Paul Leroux from QNX explained how QNX was better for Linux adopters than Linux
itself.

Needless to say, WRS wasn't going to be left behind. In February 2001, an
article appeared in
Integrated Communications Design where WRS's vice president of corporate
marketing expressed doubts about the legality of using GPL code in
embedded systems. To counter the claims, LinuxDevices.com soon published
an informational piece in which one of
the subtitles was "FUD? or a legitimate concern?". (I was personally guessing
the former.) The company's next move was to clarify its intent a little better.
In April 2001, WRS acquired BSDi's assets. It appeared that WRS was going
to counter the Linux menace using BSD. The future was to be different, however.
In October of that same year, WRS laid off its FreeBSD developers, and its
moonwalk with BSD seems to be over.

WRS still saw Linux as a threat, however. In October 2002,
Jerry Fiddler, a WRS co-founder, published a whitepaper entitled "Linux in Embedded Systems: Where
Are the Benefits?," where he essentially stated that Linux is a server OS, and that
it isn't fit for embedded applications. I won't cover each point he makes, but suffice
it to say that the copy I printed is heavily marked around the various claims.
There is, nevertheless, a hidden gem in this paper for the patient reader. Around
page 5, Fiddler makes the following confession: "... we did full Linux
development of a Tornado toolset ... We then decided not to release it because
it would push our customers into the gray area." Though he picks up the
"the GPL is dangerous" line, he clearly states that his
company has been playing at matchmaking between its tools and Linux. Despite
his other claims in the whitepaper, one is left with little doubt about the
company's continued need to look over its shoulder.

Though all of these companies have attempted to cope with Linux's emergence in
one way or another, it is interesting to note that none of them have made
any noticeable open source or free software contribution; nor have they
attempted to engage the open source and free software community in one way
or another. Their greatest mistake, in my opinion, lies in their attempt to
please embedded developers by providing substitutes or spreading FUD. The open
source and free software community is here to stay, and embedded developers'
interest in the software produced by this community is only likely to increase.

The "Embedded Linux" Market

While the traditional OS vendors were worrying about their place in the
market, a whole new bunch of vendors appeared: some of them almost overnight,
each prepared to take over the entire market.

During the initial race to conquer the embedded OS market, there were four
main contenders: Lineo, LynuxWorks, MontaVista, and Red Hat. I've already
covered LynuxWorks in the previous section, and I will therefore skip it in this
one.

In contrast to the vendors in the previous section, quite a few "embedded Linux"
vendors have collaborated in one way or another with the open source and free
software community, much like their mainstream Linux distribution counterparts.
For the community, this is an upside of these vendors' involvement with Linux.
(I've already discussed some of the downsides in my last article.)

I don't intend to provide a history of these vendors. Instead, I'm more
interested in these vendors' prospects and practices in the market.

As with all other vendors relying on the sale of software under open
source and free software licenses, the most important issue to figure out for
"embedded Linux" vendors is how to make money on software that is freely
available. Even the largest of the mainstream Linux distributions have been
struggling with this issue for quite a few years now. While Red Hat seems to
be on the track of profitability, it is unclear that other vendors will be able
to do the same. And the prospect for "embedded Linux" vendors is even worse
than mainstream vendors, since they sell software to a very technically talented
crowd. While Red Hat can count on selling support contracts to large
corporations with thousands of users, there are only so many embedded
developers out there.

Not surprisingly, the established embedded systems press has been having serious
doubts about the viability of an "embedded Linux" market.

In November 2001, Jack Ganssle wrote a column entitled "Is Embedded Linux a
Bust?" where he discusses
his doubts about Linux's ability to gather any interest in the embedded world.
The main problem with this article (and that of many others, I should add), is
that it makes no difference between Linux's viability as an embedded OS and
the viability of the "embedded Linux" market. And sure enough, some of Jack's
readers took notice. Shane Nay wrote in reply: "There are very few people that
... truly understand Linux embedded development. If you are looking to
MontaVista, Bluecat, or Lineo, you've already shown that you don't understand
Linux embedded development." Elsewhere, Neil Horman wrote: "... Linux is still
new and strange to the men and women making the big decisions. And the marketing
groups from the companies that charge 50K for a yearlong OS license aren't
clearing anything up for them," a feeling echoed in my previous piece.

In January 2002, Michael Barr, editor-in-chief of Embedded Systems Programming,
wrote a piece entitled "Open Sores," where he said: "... the market isn't growing
as rapidly as most analysts predicted it would. And I don't think it ever will."
He was probably right, but, as I said earlier, the failure of the "embedded
Linux" market doesn't mean the failure of Linux as an embedded OS. Tom Williams,
editor-in-chief of RTC Magazine, followed suit by issuing a caveat emptor to
"embedded Linux" vendors in May 2002 when he wrote in an editorial for
LinuxDevices.com that "the siren song of Linux has lured many brave hearts to
sea with a promise of treasure that is truly there. But many brave hearts will
sleep in the deep and only a few will bear home the prize."

EETimes was not missing out on the action, either. In quite a few articles, it
quoted analysts and experts calling into question the viability of the market.
In May 2002, Jerry Krasner, a VP of market intelligence for Embedded Market
Forecasters, was reported saying: "Because Linux is a
giveaway product, vendors have to sell services or tools along with it ...
and no one has yet proven that embedded developers will want to pay the cost
for those services." In July 2002, Krasner was again reported
saying: "Services are limited because they tend to be people-intensive. It's a
model, but it's not a billion-dollar model. No company is going to get really
big using a services approach." In that same article, Paul Zorfass, a senior
analyst for International Data Corporation, was stating what all open source
and free software developers already knew: "The success of these companies is
not necessary for the survival of Linux. Linux could still be successful
without them." Again in February 2003, EETimes was repeating the diagnosis: "[analysts]
are not sure a service model, under which suppliers pay to help to integrate
Linux into customer products, will prove a strong foundation for large-scale
growth."

Such conclusions are not only substantiated by argument, but surveys such as the
one recently published by LinuxDevices.com clearly show that most
embedded developers prefer not to rely on prepackaged software providers.

For all the noise "embedded Linux" has made in the press, however, one is
forced to notice that open source and free software developers who take part
in developing the technology have been completely ignored. The pundits may
rave and rant, but if they have no contact with those driving the technology
(and for sure, these aren't the vendors), then how can they be expected to come
up with a complete picture?

There is a lesson to be learned here by the established press covering the
embedded systems field: unlike traditional embedded OSes, Linux is driven by a
very active community made up of very different people with very different
interests. These are the folks making the real news about Linux's use in
embedded systems.

The Aftermath

Back to the vendors' adventures in "embedded Linux" land, pieces have fallen
through, as expected. After bleeding employees and divisions for quite a few
months, Lineo, one of the initial contenders, was finally purchased by
Metrowerks in December 2002. Red Hat, for its part, has apparently scaled down
its embedded efforts in order to concentrate on its specialty, servers.

Those that remain in the market have tried adopting various mechanisms of
survival. Some "embedded Linux" vendors, for example, require that you sign NDAs
or other restrictive licenses before they provide you with any of their
software. The compatibility of such contracts with the GPL is, of course, an
open question, given that the GPL explicitly states that no other restrictions
can be placed onto GPL software in addition to the GPL itself. And sure enough,
one such restricted license, in this case MontaVista's, was called into question
in a thread on the Linux Kernel Mailing List (LKML). As with other
threads on the LKML, there were those supporting the company and others worried
or angered by it. While I won't cover the entire thread, there is, I think, one
comment of particular interest, made by Oliver Xymoron: "Let me point out that I
never saw the NDA in question but said coworker was sufficiently intimidated by
it that he was unwilling to give me a copy of the kernel and gcc sources because
of it." One could reasonably argue that this is the desired effect of such
licenses, even though it may be legally demonstrated that these licenses are
somehow GPL-compatible.

Some vendors have also been trying to restrict the number of people to which
they have to provide copies of GPL software by invoking section 3, and 3-a
in particular, of the GPL, which requires that if you distribute a binary
that you must fulfill one of three requirements, the first being that you
provide every recipient of the binary with the full source code. Basically,
they claim that they are not required to provide the source code to anyone
else than the recipient of the binaries. While this applies readily for
unmodified code, most proponents of this argument forget to consider
section 2, and 2-b in particular, which states that there are three
requirements which must all be fulfilled in case of modified GPL software, the
second of which requiring that any published modification must also be licensed
to all third parties under the terms of the GPL. Though I don't intend to
entertain a debate about the GPL's meaning because I'm not a lawyer, I doubt
the interpretation being made by these vendors is within the spirit the FSF
has tried putting in the GPL.

Vendors are, however, not alone in trying to restrict access to GPL code.
Some of those manufacturers putting Linux in embedded systems seem to have
overlooked reading the GPL. In January 2003, EETimes was presenting a new
initiative by Japanese electronics companies to standardize Linux for consumer
devices. While discussing the matter of "securing" the device, the article
states: "The issue is how 'to not expose certain functions' to the
open source community to protect a CE device from hackers." One is left
wondering why a device needs to be secured from those developers who made the
software that it runs. In another instance, Kevin Dankwardt reports in an
article for LinuxDevices.com that when he requested a copy of the Linux software running on a device he had recently
purchased, he "was politely informed that [the] product uses 'a proprietary
version of Linux.'" Examples like this are likely to increase as Linux is more
often used in embedded systems. They do, however, point to a serious
misunderstanding by those using the OS, which could come back to haunt them,
and cost them.

As one reader of my previous article succinctly
put it: "If the 'Embedded Linux vendor' is unable to contribute the code changes
back to the originating open source project, that should be a warning signal;
either they are too incompetent (not necessarily technically, but communication-wise) or [the] client's best interests are not in their best interests."

In light of the above, there is reason to continue asking, as the mainstream
press has been doing from very early on, whether an "embedded Linux" market is
even possible. What's more, there is also reason to ask whether any of the
traditional embedded OS providers will survive Linux's entry in the embedded
world. Since I can't predict the future, I will not venture into answering
these questions. One thing is clear, however: whether any vendor is left
standing at the end of the day, Linux will continue to be increasingly adopted
by embedded system developers.

Unfinished Business

There is little doubt about Linux's interest to embedded system
developers; however, it remains that Linux itself still has some way to go to be
ideally suited to all embedded applications. Don't get me wrong, Linux is
already a good fit for quite a few applications. It has, nevertheless, a laundry
list of things to be worked on.

First and foremost, Linux is not a Real-Time Operating System. The Linux
kernel was never designed for, nor is it able to provide, deterministic response
times. Some developers are able to shortcut this limitation by controlling the
exact set of applications and drivers running on a system, but it remains that
the API provided by the kernel is non-deterministic.

The RTLinux project, which was seen by many as a potential solution to this
problem, has only hindered the finding of a solution, given its originator's
patenting of the method implemented by the project and the lack of legal
clarity regarding the patent license's applicability. At the time of this
writing, the RTLinux project is practically dead and none of the those working
on the commercial offerings of FSMLabs, the company founded by RTLinux's creator
to market RTLinux, is actively contributing new open source RTLinux code, or
helping users on the RTLinux mailing list.

Instead, the leading project for providing real-time response times in Linux is
RTAI. It has an active development community, and a very active user base. The
project also includes some of the most innovative ideas about real-time and Linux
out there, including the ability for normal memory-protected Linux processes to
become hard-real-time tasks scheduled by the real-time scheduler. RTAI was,
however, hindered by two issues: its use of the patented method for running a
mainstream OS as the least-priority task of an RTOS, and the hard-to-browse
hierarchical organization of its source tree. The first problem was solved with
the introduction of the Adeos nanokernel, proposed by your author and
implemented by Philippe Gerum, which allows different OSes to run side by side
on the same hardware. The second problem is currently being solved with the
opening of an unstable tree where RTAI developers are reorganizing the
components into a more practical layout.

In addition to being useful to RTAI, it's interesting to note that Adeos could
be used to run other OSes side by side with Linux, including traditional RTOSes
such as LynxOS, QNX, and VxWorks. This, however, would require the vendors of
those OSes to modify their products to use the nanokernel's services.

Moving beyond real-time, there's the issue of standardization. Embedded
developers, as do their workstation and server counterparts, want to be able to
develop applications and have them run on as many systems as possible. This is
where a common set of APIs, facilities, and capabilities is worth entertaining.
Before I discuss this issue, however, keep in mind that standardization in the
open source and free software world works in very different ways than
traditional efforts, where there is a governing body, and rules for submissions,
licensing issues, etc. All standards that have lived long enough to be
recognized as such by the community have had one main characteristic: they were
open for all to contribute to.

In this regard, the Embedded Linux Consortium's attempt to provide a standard
was doomed to fail from inception because the Platform Specification (ELCPS) it
developed was entirely written behind closed doors, with anyone wanting to
participate having to sign legal documents. It comes as no surprise, therefore,
that many open source and free software contributors I have talked to, some of
whom's input would be critical to any standardization effort having to do with
Linux's use in embedded systems, do not recognize the ELCPS, nor are interested
by it. Because of this lack of community grounding, there is no guarantee Linux
will move in the way the ELC wants. In fact, it may move in ways that make
any standard put forth by the ELC impossible to implement.

None of this, however, precludes the ability to provide embedded system
developers with a standard on which to build. If nothing else, embedded system
developers can rely on existing standards, such as the LSB or the FHS. Yes,
both have been tailored for workstation and server systems, but because they
are open projects, one can easily imagine having special embedded system
addendums or some similar provision for allowing the specification of embedded
Linux systems.

I have tried to initiate a move towards some form of open standardization in my
book by providing a worksheet that allows developers to fully describe their
system for others. The success of such efforts, however, relies heavily on a
common decision by practitioners to adopt a unique set of rules. Because the
open source and free software community is structured as a meritocracy, it more
or less guarantees that the best proposal will be the one most widely adopted.

There are also other issues, though smaller in scope, which merit some
attention. In particular, as I note in my book, Linux's support for industrial
networking protocols is in its infancy at the moment. Similarly, Linux's
support for being used as a USB device is embryonic. There is also the
matter of embedded GUIs. As on the desktop, there is no standard GUI for
embedded Linux systems. Time will show which one becomes dominant.

Linux has always been, and will continue to be, a work in progress. Its
successful use in any field of application has always depended on one main
thing: the active involvement of a few practitioners of that field of application
to pull on Linux to fit their desired purpose. Embedded systems are no
different, and Linux's successful use in embedded systems will be mainly
dictated by embedded system developers' active and direct involvement in its
development.

Summary

In the end, the onus is on embedded system developers. They are the ones who are increasingly deciding not to purchase traditional embedded OS vendor products, and they are the ones who are
unconvinced by the notion of paying for a freely available OS.
As I said in my book's preface, there is a clear opportunity for these
developers, many of whom are very talented, to be active and direct participants
in Linux's development. Such an involvement would bring about a best-of-breed
embedded OS that is understood and used by all as the standard OS for all 32- and 64-bit
embedded systems.

Karim Yaghmour
is the founder and president of Opersys Inc., a company providing expertise and courses on the use of open source and free software in embedded systems.