The Next Bang: The explosive combination of Embedded Linux, XML and Instant Messaging

As potential soars for the collaboration between embedded Linux and instant messaging, we are bracing for a deluge of new applications and services. More importantly, we stand on the threshold of a new way to think about--and use--the Internet.

By Doc Searls

Note: this is a longer version of the piece that appears in the September Issue of Linux Journal. It contains graphic elements present in print but missing in the online version.

6 July 2000

Linux. It's not just for computers any more.

That's the message Linus Himself has been giving us ever since he beat a career path to the Mobile Market when he went to work for Transmeta in 1997. That market is just the computerlike edge of a much larger category of devices that run on "embedded" or "real time" microprocessors and their operating systems.

Lately that market has developed a huge appetite for Linux  so huge, in fact, that it's not hard to imagine Linux quickly becoming the commodity embedded OS. Here's why:

Open source code

World's largest and most active Open Source development community

Mature, proven and still evolving rapidly

Small size--one embedded version fits on a floppy

Modular and easily customized

Net-native

Many development tools

Free--no runtime royalties required

A hot topic that gets lots of press coverage

Even though Linux is not a real time OS (RTOS), changes in the hardware world make it much more desirable in many cases than traditional RTOSes. The traditional RTOS worked in a world where hardware was  by today's standards  slow, large, expensive and specialized. An RTOS had to make the most of many different 4, 8 and 16 bit microprocessors, low clock speeds and tiny memory footprints in relatively large packages. In addition, an RTOS had to be crafted precisely to an application.

In the RT world, applications were usually in highly isolated markets. Even in the home, just about every potentially "intelligent" real time application belonged to a very isolated professional specialty that had little if any interest in other specialties. Home security, outdoor irrigation, fire safety sprinkler systems, heating and air conditioning, kitchen appliances, entertainment systems, indoor and outdoor lighting, telephones, computers and networks... these were all served by different businesses with different kinds of expertise. For proof, try to find a single contractor to install (much less service) telephone, security, computer network and home entertainment systems. The Net was never a factor for many of these categories, so it was pointless to think of all of them in a connected context. Their vendors didn't, even if their customers did.

Now connected-context is increasingly the point. "People are going to have to think differently here  even people already in the embedded space," says David Rieves, Director of Product Marketing at Lineo, one of the leading embedded Linux companies. "Now 32-bit hardware is cheap commodity stuff. So you hold overhead down in the OS, which is why Linux is so appealing. As hardware costs go down, interest in Linux goes up." Especially for devices that live on the Net.

This means we need to start thinking about real time development in a network context. While the network context is old hat for Linux, real time is not. The most familiar network services  e-mail and Web  are both essentially store & forward concepts. But to a growing community of open source developers, the missing piece here is instant messaging (IM), which by definition is much closer to "real" time.

IM is a familiar concept to the tens of millions of AOL customers who use AIM (AOL Instant Messenger) or ICQ, which AOL also owns. Neither is open and deployable like Sendmail and Apache, for the obvious reason that AOL owns the services and enjoys keeping them to itself. Even if you can create your own clients, the servers are still AOL's. They control it all. As a result, most of the world still understands IM in AOL's terms. This is why a huge percentage of IM usage is what one wag (yours truly) once called "fourteen year old girls cheating on their homework and talking about boys in real time." Worse, AOL's breed of IM is limited by its business model, which is all about selling your captured eyeballs to advertisers.

Once again, geeks are fixing the problem.

As usual, it started with a guy looking for a way to scratch his own itch. The guy in this case was Jeremie Miller. In 1998, Jeremie was working for an ISP when he started to think about IM as a UNIX solution:

"I found that more of the people I knew were fooling around with AIM, and I ended up having a need to have all of my UNIX utilities, all of my logging stuff, a lot of the UNIX environment  wanting to use this new buddy list interactively in real time...as a channel to feed my stuff through. Later on I saw there were open source libraries to get back into ICQ and AIM and stuff... and I thought about it for a couple weeks. The question was, 'What's the best way to do this?' I didn' t want to build my own client, because I can't do GUI stuff at all. I can only do back-end server stuff. I thought 'Well, I could build a back-end server, and make my own protocol, maybe I can convince somebody else to write a client, and then they wouldn't have to update it when I want features. Then I just start adding all the features into the server. Then I can bridge that server to my UNIX things that I want to add in... some logging announcements and any sort of little things I want to push up into it. I can have that server bridge over into pagers. I can have it bridge into ICQ and AIM so I can talk to my friends using other clients. And it just grew from there."

The result was Jabber, the first open source instant messaging platform. A cadre of developers quickly gathered around the project  their site is Jabber.org  and built instant messaging that any geek can deploy in the familiar manner of Sendmail and Apache. Here is a description of Jabber's architecture, condensed from the Jabber Technical White Paper :

Every user interacts through a local server that transfers messages to and through any number of other servers, each with its own domain

Jabber Identifiers are also expressed like email: yourname@domain.com

Clients and servers converse among themselves through XML streams. In client/server conversations, the XML stream is always initiated by the client to the server

The architecture can also support simple clients (e.g., a direct telnet connection) as well as AIM, ICQ and other proprietary clients

Since it is built on XML, Jabber is extensible and able to express just about any kind of structured data

Jabber's own protocol consists of XML fragments passed over XML streams between clients and servers. There are three primary protocols that define the basic types of XML fragments used in Jabber:

Messages

Presence

Info/Query

Server-to-server communication involves routing these protocol elements over an XML stream from one server to another (there are no special server-to-server protocols or features)

A Service API allows integration of security, special connections for alternate clients, and message logging

Transport servers are used to bridge the Jabber protocol to other services, such as IRC, ICQ and AIM

Jabber is built so wide open that it's hard to see the limits of what can be done with it. But that scope has less to do with IM than with XML (Extensible Markup Language). Again, Jeremie:

"As soon as I found out about XML, I kind of saw it as doing to the IT industry or to the Internet the same thing that silicon chips first did for the computing world  moving from the transistor into ICs. It's almost the same kind of phase. XML feels like that large of a change. It's morphing all the underlying structures, speeding everything up, enabling everybody to structure data in a way that other people can understand. Even if you don't understand some of the pieces of the data, you can at least look at the data and see the structure of that data. So XML was always part of that server I was building in the background. And I was using XML to push it onto the client.

XML was created to package and exchange structured data  files, blocks of text, drawings, transactions, settings, whatever  on its own terms, outside the context of the programs that produce them, and (most importantly) outside the API and protocol walls that keep services from interoperating over the Net. XML provides the basic rules for organizing content, but not for identifying it. So, while XML resembles HTML in its use of tags, it differs radically from HTML by not forcing a meaning for those tags on the data itself. Those agreements are up to those communicating within an XML framework. As for devices, that can include anything.

XML's advantage is that it's text, plus a structured dynamic framework for what you put in that text. So you can add an XML tag to control anything from an industrial valve to a point-of-sale terminal to a light switch in your home. A novel feature of XML is the DTD  the Document Type Definition. With DTDs, you get to specify document type from inside the document. This allows an infinite variety of tags. The real estate industry, for example, could create a tag called "price." Everybody in the real estate trade can know what that tag means, and program their applications accordingly. Anybody can build a DTD and publish it, which makes the framework especially flexible. It also puts control in the hands of developers working on real world projects, and not just standards bodies or large vendors with inflexible market mass.

Perry Evans is perhaps best known as the founder of MapQuest. But lately, as President & CEO of Webb, Inc. <www.webb.net>, he and Webb have taken a greater interest in Jabber than any other company to date. After learning about Jabber, Perry decided to fund its development. Webb now pays Jeremie and other core Jabber.org team members to work full time on the open source project. Webb also created Jabber, Inc. <www.jabber.com>, to explore commercializing Jabber in ways that leverage and reward its open source development (and developers). the President & CEO of Webb, Inc.

"It is extremely significant that Jabber adopted XML as its transport technique," Perry says. "Effectively allowing conversations to incorporate structure  placing documents, applications and message mining in the middle of a conversation. This plants the seeds of a whole new generation of enterprise, mobile and embedded application possibilities. We use the term "Connected Messaging" to label the way Jabber becomes connected into customer service, business exchange connections, device-based applications  or whatever. Jabber opens up all kinds of possibilities for Net-based conversations between companies and their customers, business partners and mobile employees. Plus the entire world of connected devices with a reason to traffic in live messages."

For Linux developers, this opens the world in two directions: 1) linux now operates in, and drives, virtually any kind of device; and 2) protocol-free and dynamic relationships can be created and improved on a constant basis. Of course there are enormous directory and security issues left to solve, and those are just two of the most obvious issues. But now they begin to appear much more solvable

Craig Burton, the network guru who guided Novell to success in the Eighties and did the same for The Burton Group in the Nineties, sees XML as a development as significant to our time as calculus was to the Renaissance: where calculus gave us a dynamic mathematical model, XML gives us a dynamic protocol framework  one that allows us to circumvent protocol bottlenecks by making communication independent of them:

"XML provides the framework for discovering accessibility  or a protocol  to a service during initialization. Because XML exists, developers can design communications independently of static protocols. With a dynamic protocol framework, independent  or discrete  services can exist and innovate at their own pace without the limitations and boundaries of a fixed protocol. The freedom that a dynamic protocol framework will give to innovation and implementation of Internet infrastructure will have an immeasurable impact on the future of network computing, as we know it."

Craig's idea of XML as a protocol framework makes IM a potentially critical internet service for every logical entity with a reason to exchange messages of any kind in "real-enough" time, rather than just for a few million people chatting with their buddies.

Since Linux is in an ideal position to become the universal OS on which "embedded" IM will run, let's take a closer look at where this new industry is headed.

Apply Anything

Real Time and Linux have been acquainted for a long time. RT tasks have been implemented as kernel modules in the manner of device drivers. In the practical sense, RT tasks can be developed and implemented in Linux user space where they cannot overwrite critical areas of kernel memory.

But for real-time precision with minimal interrupt latency, more dedicated solutions need to be found. Victor Yodaiken and Charles Brabanov's approach was embodied in RT-Linux, which was first released in February 1996. They didn't think working along Linux' POSIX lines would deliver that. So they created an emulation layer that insulates the upper layers of the OS from interrupts, intercepting those interrupts on behalf of the OS.

In this scheme, when Linux thinks it turns an interrupt off, it sets in a flag in the emulation layer. While interrupts stay on, the emulation layer intercepts the interrupts that occur while Linux believes it is uninterruptible. Intercepted interrupts are queued, so no interrupts are lost. But the real time work is done by a deterministic adjunct real-time kernel. This division of labor essentially creates a real time Linux OS  except it's not really Linux where the rubber meats the road.

More recently another version of that approach has emerged. RTAI (Real Time Application Interface) also has a separate real-time module that operates on top of the Linux kernel. Lineo is the prime developer for RTAI, and claims to have contributed the majority of the RTAI code base. "We recommend it RTAI because it is a cleaner design, has equivalent performance, and includes enhanced features. With Embedix, hard real-time tasks are written with either native RTAI API or with the POSIX 1003.1c API," says Lineo's Jennifer Finlinson. She adds:

"When you implement a hard real-time task, you can remain essentially unconcerned with anything else that might occur on the computer. External and on-board events are serviced with lower priority than the critical task. We believe the developer needs support for both hard and soft real-time. Many of the reasons people pay for expensive proprietary RTOS solutions such as pSOS and VxWorks is because they want hard real-time.

"RTAI features should only be added only as they are needed. Minimal real-time scheduling services take up 65 Kbytes of additional code space. If you add everything in you get to about 300 Kbytes."

That's just one approach, but it's the one that speaks directly to the inadequacies of Linux in the real time world. Here's IDC analyst Dan Kusnetzky's overview of the embedded Linux market as it now stands:

"Many organizations are working to find ways to use Linux as an embedded operating environment. Red Hat, and Lineo are only the most recognizable names working on this. Some are interested in providing Linux for internet appliances. Some are focusing on server appliances. A few are focusing on Linux as a real-time operating environment. The last is quite a trick because the Linux kernel doesn't have the features for the most intensive real-time applications.

"Some focused on real-time Linux have developed their on real-time kernel to replace the Linux kernel. Others have Linux boot up their own proprietary real-time environment. Although the run time environment isn't Linux, they can talk about real-time and Linux in the same sentence because Linux is what people see when it boots up.

"IDC is expecting the Linux community to settle on standard approaches to support both embedded and real-time applications. Until then, we have a number of competing offerings from different suppliers."

The competition is heating up fast. James Ready, the embedded pioneer who founded Ready Systems a few years back, now runs MontaVista <www.mvista.com>, which launched HardHat Linux last year. RedHat acquired Cygnus last Fall, then added WireSpeed in June of this year. (The sidebar is by Michael Tiemann, the Red Hat CTO who founded Cygnus.) Last summer Caldera, an early Linux distribution leader, spun off Lineo as an embedded Linux company. Both Lineo and MontaVista have substantial venture capital backing, and are themselves in aggressive acquisition modes. With all this movement in a short time, traditional embedded publications and events are suddenly abuzz with Linux coverage.

And traditional RTOS isn't going away quickly. Wind River (which makes both VxWorks and pSOS) is the market leader, with around $200 million in sales and a multi-billion dollar market cap. "Wind River is the 2000 pound gorilla in the embedded OS market," says Bryan Sparks, President & CEO of Lineo. "But we're finding a lot of interest in moving away from Wind River products. If we can support the same platforms that Wind River does, we can take more and more business away from them." In addition to Intel, AMD, PowerPC and other ususual suspects, Microprocessor platforms include Hitachi SH, MIPS, ARM and StrongArm, ndcore, ColdFire, Dragonball and others. "We can get Linux on all those platforms," Sparks adds.

Some of those platforms are very small microcontrollers that dont have full-featured CPU sets. Typically they lack an MMU (Memory Management Unit). So Linux needs to be modified accodingly.

Why embedded Linux?

A happy consequence of the open source model is that Linux evolved, by necessity, to be more modular, and to support cleaner interfaces, than a monolithic design that assumes less anarchistic control. As a result, Linux kernels as small as 300KB can not only boot up with networking support, but serve web pages as well. This is really exciting to people who want to put web servers (and administrative consoles) everywhere.

You can also use the same APIs for development and deployment. While this is not a new goal, we now have a system that (1) works, (2) people actually like to program, and (3) supports tons of content. This has expanded the realm of embedded system development from a highly specialized art to something any Linux programmer can manage. This is going to have a profound impact on the embedded system market.

 Michael Tiemann, Chief Technology Officer, RedHat

The move to Linux is happening most rapidly around mobile systems. Intel is seeing strong Linux support for StrongARM. Compaq has created the Open Handheld Program, and released a Linux version for its iPAQ handheld PC. The company also sponsors Handhelds.org, which supports open source software development for handheld devices. Given Linux' popularity as a hermit crab on x86 boxes, Windows CE handheld devices are an obvious embedded Linux target. While Palm charges more for a software license (around $20/unit) than Microsoft charges for Windows CE (around $10-$15/unit), Palm's units are more arcane and less powerful than typical Windows CE devices, so moving Linux to the latter is a shorter step. This reportedly is HP's approach. Boris Elisman, worldwide marketing manager at HP's information appliances and services division, says the company is developing a number of multifunction Linux devices that should come out in 2001.

Designing embedded systems around Linux is an extremely easy choice for OEMs to make. Here's John Bork of Intel, which is building its new set-top boxes and Dot.Station Web appliances on a combination of embedded Linux and customized Mozilla:

"We just take the source off the Net, pull out the stuff we don't need, seeing what libraries are used and not used, and constraining ourselves from there. See, the biggest challenge we face is getting the OS, the browser, the plug-ins and a few applets into eight megabytes of flash. So a lot of our work has gone into analyzing what pieces we need, and how we skinny this thing down."

This work has a lot of appeal. "I enjoyed working on servers at my last job," says one Linux programmer at Kerbango, a startup implementing MontaVista's embedded Linux in its Web radios, "But I love working on cool new stuff here." In the past, learning a whole new operating system and all its tools was expensive and time-consuming. Not so with Linux. An embedded application might be extremely arcane, but the operating system knowledge required is not, because Linux and its tools are so familiar to so many developers. This fact can hold down NRE (non-recoverable engineering) costs, which are always critical factors.

Figure 1. Embedded Linux Appliances

Kerbango's radio and Intel's Dot.Station are just two among the first wave of Net-native appliances built using embedded Linux (the Dot.Station also uses Mozilla).

"Skinnying down" Linux, however, is not an easy job. That's why companies like Lineo and MontaVista are in business. Here's Kim Clark, VP Engineering at Lineo:

"The first thing you need to understand is, what are all the packages? Which ones do I need and which ones don't I need? What are the dependencies associated with each of them? Choosing one package may require choosing two or three other packages. But they also might conflict with packages from another module. So you have this hairy and complicated task of figuring out all the things that need to be there."

Among Lineo's tools is one called "LIPO," which goes in and, well, sucks out all the fat. "The shared libraries are big and fat," Clark says. "If you have several modules sharing libraries, each with a copy of that library, you've got a bigger footprint. We cut fat out of that. We also reduce the size of each library's' footprint. We do the same for Apache. By cutting out icon support, for example, you save 22k of memory. We make this possible by taking Apache, cutting it into microcomponents, giving developers the choice of using only the parts they need." (See Figure 2.)

These screen shots shows Lineo's tools at work, selectively enabling parts of both Linux and Apache.

Using tools like Lineo's helps Linux apply across a huge assortment of embedded devices, as we see in Figure 3.

Figure 3. Lineo's "Embedded Linux Market Spectrum"

This chart differentiates between the space where the company's embedded product offerings (and acquisitions) apply, and the non-embedded PC/Server space, where they don't.

This is Lineo's view of the world, but it could be anybody's. None of the lines are definitive, which is why the boxes are shaded.

Lyle Ball, VP Marketing for Lineo, says "The important thing is that, while the desktop and server areas are familiar territory, the rest of the world  the embedded device world  is not. There is a lot of existing expertise everywhere on this chart, but when you get down to the finished product level, that's where you've got the feudal system that Linux and these other developments begin to eliminate."

Real World Real Time

Behind every cell in Figure 3 is the Net. And behind the Net's popularity is the growing expectation that just about everything will eventually be connected to it, and communications over it will be instantaneous. This is the real world meaning of "real time." In a fully distributed computing world, any device should be able to communicate with any other device in real time  or close enough. What's the infrastructure for that?

The Jabber People think it's XML-based Instant Messaging. IM by itself is about two functions: presence and messaging. The first involves detecting and revealing the availability of another identity on the network. The second involves one-to-one (or one-to-few) live text exchanges with one or more present members of a list.

In enterprises where IM is widely adopted, life isn't the same for other technologies. "It changes the sociology of telephony," says Udi Shapiro, an instant messaging pioneer whose company, Ubique, created an IM system that was owned for awhile by AOL before Udi and his partners bought it back and sold it to Lotus, which now offers it as SameTime. According to Udi, IM in corporate settings becomes an accessory to the telephone, replacing brief or trivial calls in some cases and initiating them in others. The next step would naturally be to meld IM and telephony technologies: in wired telephone systems; in Voice over IP (VoIP); in wireless communications between handheld devices; and in combinations of all those, plus Web, email and other Net-native communications.

It's not hard to imagine what IM can do for communications between humans. The harder but necessary step is to imagine communications between humans and remote machines, and among machines themselves, on an ad hoc real time basis. This is where embedded Linux comes in.

When IM is combined with Embedded Linux, IM provides the framework for registration and lookup, while embedded Linux works as the device driver in the API. Through the instant messenger, the client says to the server "I'm online," and the server notifies other clients of that fact, providing the path for direct communications. Thus instant messaging becomes an ideal way to set up point-to-point communications between two unknown entities  human or otherwise.

The best working example of this arrangement is cellular telephony. The problem with that example, however, is that cell phones are physical devices. The system doesn't know the identity of the user  only the phone number and the unique identity of the phone it is assigned to. With IM you can locate the person or the identity of the device. In either case, it's a logical connection and not just a physical one.

IM is also similar to cellular telephony in its real-time nature. If you think of embedded messaging as a service, it makes no sense to have a store & forward architecture, because you are working mostly in real time. A real-time device doesn't have the peripheral support (such as storage) requried by a computer. There's no point to it. So, for live communications, a real time connection can be established through IM.

With Jabber, XML messages become an envelope or container (see Figure 4). The contents of that envelope are infinitely flexible. XML's data is "structured," but also potentially dynamic  able to change constantly. It can even define the terms by which it is understood.

Figure 4. One Jabber view of XBIM  XML Based Instant Messaging: XML as a container  or "envelope" for pretty much anything

XML is designed to carry "structured" data. But that data can change constantly, and  by design  can define the terms by which it is understood. In essence this creates a standards-based framework for dynamic relationships between communicating entities on the Net. When those communications become "instant," there is nothing to stop relationships from being defined and changed whenever necessary.

When communications become "instant" (which in practical terms means near real time), there is nothing to stop relationships from being defined and changed whenever necessary, and by agreement between the communicating parties, even if those parties are intelligent devices. With Jabber, any two identities, whether human or machine, can send and receive real time messages that contain pretty much anything, and do so in a structured way, independent, if necessary, of intermediating protocols.

Although Jabber was originally envisioned as a distributed, XML aware instant messaging platform, its uncomplicated methods of combining presence with XML document routing puts it in an ideal position to become much more than just another instant messaging system. instead, it could become infrastructure for messaging enabled embedded applications. It becomes the way they communicate through the Net  whether with each other or with apps on the PC (which are already talking to each other).

Figure 5 shows a Jabber client/server platform built for the purpose of routing XML data in real time. Note that the underlying messaging infrastructure and protocol is XML, and not a proprietary messaging protocol. Messages can be sent from device-to-device, device-to-PC, or translated from any of the above to other networks and protocols. The dynamic, self-determining nature of XML makes this infrastructure dynamic: in other words, to accommodate constantly changing relationships between logical entities on the Net.

Figure 5. Jabber as a real time messaging platform

Jabber is an instant messaging platform with the ability to packetize messages in XML documents  in effect as data transport. Any PC or Net-native device, can instantly communicate with all of the infrastructure on the Net that can parse and act on XML. This puts new XML databases  which are designed to parse XML and store structured data or documents  in a powerful context. Anywhere XML is the preferred way to receive and send information, we have infrastructure already conformed to instant messaging.

In this sense Jabber is a ubiquitous XML router, streamlined for (and by) embedded Linux  or any other device-driving OS that supports XML-based messaging. Real (enough) time is the key. While email is technically capable of routing XML documents, it lacks awareness of presence, which is key to Instant Messaging in general. This makes Jabber something of a new message transport breed.

This breed will expand Internet infrastructure to accommodate a vast new range of dynamic real time relationships and activities, routed through XML. This invites the development of suites of robust client-side libraries on multiple OSs, gathering around embedded Linux. These will constitute an SDK for connected messaging to the device world.

In fact, this kind of development is already taking place. Lineo recently decided to fund Jabber development, and to include Jabber in the company's Embedix SDK.

Figure 6. XML developments In the context of device relations across an XML data stream

This shows shows how new and emerging tools (like XML optimized databases) can make use of (and be used by) XML communications streams between devices, and to provide new services. The fact that all communication with Jabber is via XML documents means that all new XML tools and services can be easily 'plugged' into the message stream carried by Jabber.

"Emerging tools (like XML optimized databases  see Figure 6  ed.) can be used to leverage XML communications streams, and to provide whole new services," says Andre Durand, General Manager of Jabber.Inc. "The fact that all communication with Jabber is via XML documents means that emerging XML tools and services can be easily 'plugged' into the message stream carried by Jabber. Extrapolate that, and you've got a device driving framework for the known world. Because it is now quite simple for you, or a contractor you hire, to program your home  say through X10." (See sidebar.)

Big Bang 2.0

What Jabber does with IM, XML and embedded Linux adds huge new conversational territory to all three topics. Inside that territory is  whatever we want to blow up.

Maybe we're wrong, but we think this is where the Next Big Bang is going to happen. The last time we felt this way, "we" consisted of Phil Hughes, and the spot we watched was Linux. That was in 1994, when both Linux and Linux Journal were pre-1.0.