Embedded Linux: Semantics and Reality

Let's put it bluntly: embedded Linux doesn't exist. Embedded Linux is the stuff of glitzy announcements, hype, and other marketing mumbo jumbo. That is, at least, the conclusion I am forced to reach after having spent two years writing a book about the use of Linux in embedded systems, which is an entirely different subject. For had I written a book about "Embedded Linux," it would most certainly have been prime material for Marketing 101.

To understand why I make such statements, let us explore the historical and contemporary uses of "embedded Linux". Among other things, we'll tackle a question I find quite interesting: Is "embedded Linux" a noun or an adjective? Sounds silly, right? Well not quite ...

The Roots of "Embedded Linux"

One of my favorite tools for conducting an anthropological study of trends
on the net is Google Groups. It's fascinating how one can select only those
messages between a certain date and another, or the use of certain keywords and the absence of others. The data collected in this way can then be
used to analyze the surge or drop in interest for certain subjects against
actual events and trends.

On the subject of "embedded Linux", my Google Groups search was quite
fruitful. For example, I was able to track down one of the oldest, if not
the oldest, reference to the compound use of the words "embedded" and "Linux."
The message entitled Embedded Linux kernel anyone?, sent to comp.os.linux by Bustamam Harun on the 22nd of February 1993 is of the "has anyone done this before?" type. The posting's title is quite revealing. Bustamam didn't ask about "embedded Linux," he asked about an "Embedded Linux
kernel." Somehow, he found it important to mention that he was looking for an
embedded version of the "Linux kernel," not just Linux. It would, therefore,
seem that the first time it was ever used, "embedded Linux" was an adjective,
not a noun.

Later discussions on the various groups seem to follow that same use. Take
for instance the thread that starts on the 25th of October 1994 on
comp.os.linux.development with a post from Trond Solem entitled Embedded Linux. This thread is interesting, as it includes some of the first instances that
I can find of kernel developers discussing the use of Linux in embedded
systems. In particular, Alan Cox and Ingo Molnar participate in the thread.
Though the word "kernel" isn't used here, the thread quickly centers around
kernel issues and there is no hint in this thread that the "embedded Linux" noun may mean anything else but the kernel.

Other messages sent during that same period in 1994 have titles such as
ROMmable Linux? and 68360 linux. Even later in August 1995, when someone from CompuServe posts a message to comp.os.linux.misc entitled Embedded Linux in EPROM?,
he or she asks: "Has anyone else tried to embed the LINUX kernel?" Again, the
discussion revolves around the kernel.

In the following years, the compound use of the words "embedded" and "Linux"
remains rather marginal until 1998. Here are the number of messages containing
instances of the compound term "embedded Linux" and the loose words "embedded"
and "Linux" found by Google for each year since 1992:

[1] In some cases, the use of the word "embedded" in the same posting as the word "Linux" has nothing to do with embedded systems.

As late as 1998, "embedded Linux" is still quite often used as an adjective and
there are many instances of "embedded Linux kernel" and "embedded Linux system."
Even when used as a noun, "embedded Linux" quite often designates something
specific, such as a distribution or the kernel itself. However, one can
observe an increasing trend of the use of "embedded Linux" as a noun without
reference (that is, "embedded Linux" as being a thing in itself, not a distribution,
a kernel, a system, or anything else.)

The Invasion

Starting in late 1999, the year of the great Linux hype, things start to get
very confusing. We start hearing about "embedded Linux companies," "the
embedded Linux market," an "embedded Linux journal," "embedded Linux hardware,"
"embedded Linux sites," etc. In contrast, Google Groups is unable to find any
reference whatsoever to "server Linux companies" or the "workstation Linux
market" in its entire archive. Somehow such terms sound odd, but their "embedded
Linux" variants have become mainstream. Even the straightforward "Linux server
companies" only scores one message in the entire newsgroup archive. So
maybe the word "embedded" is special? Well, it turns out that there are no
mentions of "embedded DOS companies" and but two references to "embedded Windows
hardware." It's not like these operating systems haven't been used in embedded systems before.

Nevertheless, 1999 is the year of the Linux phenomenon and that year, many
companies came forth promising to rid the world of those evil proprietary
embedded OS vendors by helping embedded developers use Linux in their products.
The moment these companies entered the market, the term "embedded Linux" lost
all meaning. It became, as I said earlier, a meaningless mishmash; the stuff of
marketoids.

The Desert

Anyone after the invasion period trying to really understand how Linux can be
used in an embedded system found himself trapped in surveying the products of a
few select companies, or roaming the vast dry desert of those wanting to put
Linux in an embedded system without having to rely on any vendor whatsoever.

In essence, while the number of people, magazines, sites, and guides dedicated
to "embedded Linux" was certainly on the increase, the signal-to-noise ratio was
getting very low. To escape the craziness, many developers roaming the "I want
to understand" desert found their way to various mailing lists related to the
use of Linux in embedded systems. There, many built their expertise and finally
got the point where they were able to make it on their own and help others in
reaching similar goals. Indeed, some of the very best information regarding the
use of Linux in embedded systems is still found in mailing list archives.

While many talented souls were able to make it with mailing list help alone,
many others were still left wondering where to start. Fortunately for many
vendors, the use of Linux in embedded systems is nontrivial and a few of them
prospered.

The situation was, however, far from ideal for those of us in the Linux
community who really value freedom. I personally found it rather ironic and sad
that although all of the necessary software components were available under the
terms of the GPL and the LGPL, very few people were actually exercising the
freedom to use this software in order to build embedded systems.

Among other things, I gradually developed a kind of allergic reaction to
the use of the term "embedded Linux." Indeed, every time I see it used
somewhere, I put on my "skepticism" goggles and ask myself "what's the real
subject of discussion here?" or "what's for sale?"

My frustration with the ongoing hype, and the continuing lack of proper
documentation, was one of the reasons that pushed me to write my book. Following
the open source and free software tradition, I had a clear intention not to hide
any of the most arcane tricks to the reader. I was bent on giving readers
everything they needed to make it, based solely on the original source code, and
to provide them with everything they needed to get the rest of the information I
simply couldn't fit in the book.

Where To?

While this article attempts to break new ground and elicit discussion around the use of the "embedded Linux" term, I must confess that my book provides a very
conservative definition of "embedded Linux." It remains, nevertheless, that I
wish that things change.

As a first resort, I thing we should all agree that "embedded Linux" is an
adjective. That would really help in narrowing down the subject of discussion.
There's no way you would accept to buy something from someone selling you "big
green," or "small red." There's no reason that you should buy into anything
having to do with "embedded Linux," either. You would, however, know the subject
of discussion if someone approached you about an "embedded Linux kernel," an
"embedded Linux distribution," or an "embedded Linux system." This is why the
title of my book tries to be as specific as possible: Building Embedded Linux
Systems. As with any other adjective, however, there will continue to be some
uses of the "embedded Linux" adjective that make little sense. I remain
perplexed by those selling or looking for "embedded Linux hardware." Personally,
I'm usually looking for "embedded hardware that runs Linux." It usually helps me
find products rather than hype.

Secondly, I think everyone should understand that there is no special version
of Linux that runs on embedded hardware. The different kernel trees developed
for each architecture are all publicly available for download. In the case where
you need a "special" kernel whose source doesn't come through a publicly
accessible URL, then I encourage you to think twice before using it. There are,
of course, a few key packages which are more often used in embedded systems, but
these same packages can be used on servers and workstations all the same. Such
is the case of uClibc, for example. Other packages still are exactly the same as
those used on workstations and servers, but are configured differently when used
in embedded systems. This is why I completely agree with Red Hat's Michael
Tiemann when he wrote close to a year ago that "... Linux has the requisite DNA and development model to truly scale from embedded to enterprise as a single platform ..." Indeed, Linux's ability to
scale to many different hardware configurations is one of the reasons why it's
such a good choice for an embedded OS.

By exercising these two basic recommendations, I think we should be able to
reclaim the "embedded Linux" adjective and return the focus back to the
values at the core of the free software and open source community.

Karim Yaghmour
is the founder and president of Opersys Inc., a company providing expertise and courses on the use of open source and free software in embedded systems.

Community Resource

In an effort to jump-start such changes and to help developers find a community
resource on the use of Linux in embedded systems, I have set up a companion site for my book. Among other things, the site features a mailing list for discussing issues not already covered by other mailing lists. I encourage you to visit the site, to
subscribe to the mailing list, and help in building a solid technical understanding
of the use of Linux in embedded systems.