Posted
by
Hemos
on Sunday May 05, 2002 @04:50AM
from the what-will-happen dept.

Embedded Geek writes "Jack Ganssle has posted a column at embedded.com pondering whether Red Hat and other open source companies serving the embedded community are due a shakeout similar to the dot com collapse. He cites Red Hat's March cuts in their embedded division and their losses of $80M to $140M a year. He admits, though, that because the embedded market is smaller and many companies are privately held it is difficult to get a pulse on what's going on behind closed doors.
"

Right now, it's more like "Which company isn't having financial issues."

This stuff is getting rather old. Everyone is having a tough time being profitable and I don't believe "opensource" is the reason why RedHat is struggling. You name the company, products and services that were selling well in 97-00 are no longer doing that well. EMC and Sun which used to be very profitable at what they did are having to switch gears. Cisco is having it's share of problems. Software or hardware everyone is struggling and Opensource companies are still trying to figure out how to make a profit on a service model.

Folks this isn't news and really doesn't need to be argued over but Opensource isn't the next "Dot com bubble" that is going to burst. Opensource, Redhat, whatever... everything is having trouble right now. Let's not make this out to be more than it really is.

Everyone having problems now? No, microsoft is making profit, oracle also makes profit.

Microsoft and Oracle are everybody?
What he means is the economy in general, as well as most of the tech industry is experiencing a down turn. Providing exceptions doesn't change the stats quo that is happening at the moment

Red Hat's embedded division wasn't selling Linux based solutions, it was selling the eCos product that came from its acquisition of Cygnus. A minimal eCos kernal can be about 4K of ROM and 1K of RAM. [redhat.com] The kernel isn't a POSIX compliant API, (although they seem to have recently built POSIX/Linux compatibility layer on it called EL/IX [redhat.com] which would put it in the same ballpark as QNX.) I agree with your main point though, many of the big system technologies (C++, Java, Linux) seem out of place for embedded system design.

I'm sorry, apparently I chose the wrong words or phrasings, since you seem to have misunderstood my message.

The eCos kernel, the kernel that RedHat acquired when it bought Cygnus, does not have a POSIX compatible, or even unix like, kernel interface. It has a kernel interface, it just looks very different than Unix. They weren't creating an incompatible kernel just to be different, they were doing it to create a kernel with features ideally suited for the development of embedded devices.

As an example, a separate thread of execution is not started with fork() or pthread_create(), but rather a thread is created with cyg_thread_create() or sta_tsk() if the ITRON API is built into the kernel. If you look at the eCos API [redhat.com] and something with a Unix or POSIX API [tldp.org] you will notice that few if any of the same calls exist on the two platforms.
Just to reiterate, so I'm not confusing you again. RedHat's eCos is a kernel designed for embedded systems and is not at all similar to the POSIX standard.

Now QNX as a developer of of software for the embedded market took an interesting strategy. Since there is so much software written for a POSIX style kernel, and it is an API that many people are familiar with, they created a kernel small enough for embedded devices, but gave it a POSIX API. This may make some tradeoffs in many directions, since what the program needs from the kernel and what the kernel needs to inform the code is very different for embedded devices than for a desktop or server OS.

Now to my comparison between eCos and QNX. When eCos is built with its optional EL/IX component, it starts to implement a subset of the POSIX API. So calls like read(), write(), and fcntl() are available. If you flip on the EL/IX switch, it starts getting a lot more familiar to Unix developers, which getting the product a little towards what QNX's product line does. Or at least having the same advantages.

Unfortunately, I can't quite understand how you read what I wrote originally and picked out the two sentence fragments you quoted as being a summary. If you could point out where my original post was unclear, I can try to avoid making the same mistake in the future.

Finally, to directly answer your points.

No, you are right, a kernel isn't an API, but a kernel does implement an API. eCos does implment an API, but that API (without EL/IX) is very dissimilar to POSIX.

Now for a company like QNX with a product like they have, I can perfectly well understand why they would participate in an effort like POSIX. As the committee is debating the pros and cons of particular features, they really want to be there to argue against features that could be construed as adding bloat.

Being on a committee doesn't make a vendors product more or less compatible with the standard the vendor produces.

I don't think many Open Source projects will ever be considered "finished," because that is exactly the point - the source is open - thus allowing for additions/changes, and it will never be "finished".

And anyways, just what exactly does "finished" mean when talking about OSes? From the way I see it, a complete operating system would be a very difficult thing to "finish," especially an open sourced one where many, many people around the globe contribute.

There is a saying the the software development industry and that is when you have done 90%, you have 90% to go. This means that when you have implemented 90% of the functionality you still have 90% of the job left to do. There is much truth in this.

This is the problem with almost all open source software, noone is interrested in really make a FINISHED product thats usable. It's time-consuming, boring and hard to finish a product and most people aren't very interrested in beeing bored on their free time.

One problem with embedded Linux solutions is that developers are compelled to release their modifications as per the GPL. Whether or not this is a good thing (and there are some very good reasons for both GPL-style and closed-source software, beyond the scope of this comment), some companies just don't enjoy the thought of having to release the source code for their embedded products.

Embedded BSD is the solution to that problem, and in fact various forms of embedded BSD have been around for a long time and are going strong. Perhaps if Linux changed its license from GPL to LGPL it might help in this situation, to gain more acceptance from the business community.

Actually, most embedded modifications can be kept proprietary, through the modules exception on the kernel and through placing of proprietary code in a user-mode program.

I created the user-mode half of an Open Source embedded system - a program called busybox. It's everywhere. It is hosted on Linux much more often than BSD. People don't seem to need BSD licensing on the kernel that much.

The problem is not a lack of customers. Linux is the standard for embedded systems these days. The problem, if you call it a problem, is that a lot of vendors aren't profitable at offering embedded Linux. It's too easy for the customer to get along without those vendors. And I have no problem with that.

I just helped produce a *very* cool consumer device basically running a modified redhat distro. Granted, it was a set-top box with no hard real-time requirements and lots of resources (Big HD, lots of RAM), but even if it wasn't, I still wouldn't have needed much help to get going.

This is the strong point of open source - I don't need to purchase expensive service contracts and fancy software to get the job done. I have everything I need to be productive and make modifications to the system. I fully appreciate the contributions of the embedded linux vendors out there, but I really have to wonder if the service model ever made sense.

As far as licensing issues... Ugh! The horse is long since deceased, STOP BEATING IT! This is a typical scare tactic used by closed source s/w vendors and BSD freaks. Sure, as a contractor I much prefer the BSD license, but structuring things to comply with the GPL isn't that difficult.

As far as I know, Cygnus was profitable, at least in the Golden Years just before Open Source more or less hit the mainstream press. There is an interview at developerWorks [ibm.com] in which Cygnus Solutions cofounder Michael Tiemann claims that they were profitable until the venture capitalists came, since investors "give you money so you can actually accelerate the rate of spending versus revenue".

It's probably not too easy to turn back the clock for Red Hat, as there is increasing competition on the GCC customization market, where companies pay immense sums for adaption of GCC to certain microprocessor platforms and support for that GCC derivate (at least they paid these sums when Cygnus didn't have much competition!).

Simply, because gcc is the best compiler of the world!! Note I say gcc, not g++. (C++ is and had it's issues). Well best in regards of retargetability and availability of cpu targets, very good in regards of improving development code warnings and errors, good/better medicore in regards of code generation. below avergare in regards of compilation speed. Absolute first class in regards of price and per-seat licenses:o) On the top of things in regards of standard compability. (Note that there isn't a single c++ compiler in the world that meets the ANSI standards...)

Red Hat's March cuts in their embedded division and their losses of $80M to $140M a year.

Do I need to explain the concept of "Goodwill" yet again? Why do people keep this shit up? Are they holding short positions in RHAT?

Goodwill is the amount of money a company pays for another company, that is above the value of the tangible assets of the aquired company. This may include brand names, patents, and other intangibles. If a company buys a company that turns out to be overpriced later on, then the buying company will have lots and lots of goodwill that must eventually be charged against earnings.

Red Hat did not lose those large amounts of "real assets", rather, almost all of that "loss" was a write down of goodwill from previous aqusitions.

Look at it this way.

1) RHAT IPOs2) RHAT stock becomes grossly overvalued3) RHAT makes a secondary offering, cashing in on their grossly overvalued stock in a big way.4) RHAT goes on a buying spree, spending their money that they got for free, buying companies like Cygnus.5) Cygnus was also pretty overvalued, so a lot of goodwill ends up in the "assets" column of RHATs balance sheet.6) RHAT has to write off chunks of this goodwill against earnings later on.

The key is that RHAT got this money for free. Had they not made the secondary offering, they wouldn't have had the money to make the aqusitions in the first place. Sure, that money ultimately came from idiot investors that paid $200 a share for RHAT, but it didn't come from any direct business or financing activities that had an opportunity cost for RHAT (such as debt financing).

So no, they aren't losing those staggering numbers each quarter, in fact they are breaking even for the last 3 quarters or so.

Goowill can be abused. "One-time-charges" can be abused (see Cisco writing off billions of real, tangible, inventory).... but in this case... there is no money lost, just worthless monopoly money that no longer exists, and hasn't existed for years, subtracted from a column on a balance sheet.

That much is correct, but I don't think many of the people that bought in at $200 are still holding the company. Unscientific, but from the RHAT message boards I frequent, most of the posts come from recent buyers, I've not seen anyone who rode it all the way down and is still holding it. My point is that those people would have lost nearly the same amount of money whether RHAT had made the secondary offering or not.

RHAT did dissapoint in the most recent quarter, they broke even again when it was expected that they would have at least a modest profit by now.

That still doesn't give people the right to talk about losses that are attributable to goodwill as if they are operating losses, it's not an accurate picture of what the status of the company actually is.

> there is no money lost, just worthless monopoly> money that no longer exists

The lost money is real money no matter how you slice it. Part of running a business is using your money/stock effectively. By overpaying for other companies, Redhat management is showing that they are no smarter than the dumb investors you are talking about. Incompetence is incompetence and it will reflect on the balance sheet sooner or later.

So there's no reference (by name) to any other company doing embedded open source development, no attempt to determine why RedHat and other (U.S. only) companies laid off staff... Sorry, but this article is fluff.

I work for an (open source technologies) embedded development consultancy company. The only thing limiting our growth is finding competent personnel. Demand exceeds what we can deliver by a large factor. Linux has taken the embedded development world like a tornado grabbing a trailer. I see no bubble building up here.

Of course, other companies may have been less realistic in their growth perspectives and may have had more venture capital to burn.

On the other hand I had a chance to chat with a Wind River salesman for there development toolsthe other day over beer. (I don't do embeded work so this was on the casual conversation side, and he had no reason to tell me bogus stuff.) He said that he expected to sell more embedded development tools for Linux. (There tools are available.) He hadn't. No demand. Either the people using Linux embeded are not using Wind River or that is just a sign of the times.

Linux has taken a huge percentage of the "do it yourself" market which was actually about 60% of embedded. Of commercial embedded development, WindRiver used to own about 60%, but that number is going down fast. Linux is expected to win more new design wins in the next year than WindRiver, making it the #1 choice for new designs. An important thing to remember in embedded is that a large percentage of the market has legacy deployments in the market already and those will continue to be VxWorks (or pSOS) for the forceable future. But Linux is here to stay...

Sure, a number of companies will shake out. Some people blame this on the Open Source nature of the software, but they are really missing the point. Open Source is not a business, it's a cost-center that is shared among a large number of collaborators. A cost-center is less than half of your business plan. Open Source is non-differentiating for your business, because everybody else can offer the same thing, and thus the best plan a business can have for it is to use it to reduce cost so that they can spend more on their differentiators and profit centers.

In making a business plan, you will need to figure out what your differentiators and profit centers will be. It's obviously not going to be software sales - the folks who tried a royalty-based embedded system based on Linux went out of business faster, because they had free competition. In general it will be consulting services, and this is going to be a difficult business in a slow economy as businesses will try to do more with their own engineers. Businesses also have an incentive to use their own engineers for embedded work, as they don't want to be in the situation of losing the recipie for one of their own products. That can happen more easily when an outside vendor does the work.

I created the user-mode half of most Linux embedded systems - a program called busybox. It's everywhere. I used the GPL. Because of that decision, the person who put the most effort into maintaining that program, after me, is still working on it and is able to offer his consulting services on it. Had I not used the GPL, he would have had to give it up when his previous employer was one of those shaken out. I have a lot more sympathy for him than the employer. Also, had I used a license other than the GPL, the program would not have become an open standard for embedded - everyone would have been making their own proprietary additions rather than cooperating. And I didn't care that companies could not lock in a revenue capture on busybox - why should I?

I think Debian has the best "business plan" of any Linux distribution: don't even try to make money. The people who use Debian as a cost center (HP, for example) pay for its development, and they are very clear about what their profit centers are. This is why I think Debian will eventually end up on top.

I agree fully with your view. The interesting question is what RedHat has for a profit center. I always thought that selling CD:s with Linux is very close to selling Hot Dogs, they get old pretty fast. Stallman once claimed to write free, smaller and faster replacements for UNIX programs. Today this is history but not many has understood the reasons why he did it. They still try to make money from software. I think Red Hat and its alike has made a great work gathering venture cap for the open source cause. It IS a bubble and it will burst, but the open source will always remain for those who got their profit centers right.

After the bubble bursts a lot of bad open source software will vanish and we will wash out what is worth saving and build on that. An evolutionary thing I would like to say and it is not a reversable process. I vonder though why the investors views a possible future market share by open source software as attractive. I mean zero dollar one time is as much as zero dollar a zillion times. *scratch* I am probably too stupid for that game...

In my opinion Red Hat and many of their alikes are very much aware of this but while trying to think big they loose focus from "small and fast" and just keeping "free" will not make it. I have dissapointing experiences giving up installing Redhat on an old PC with a small (600 Mb) hard disk as well as trying to read a word document using Staroffice with only 64 Mb of RAM! M stands for mega (million) remember! *sigh*

Well the StarOffice you were using (5.2) wasn't open source software. It was available fro free but wasn't open source. So it's not a very good comparison really. Try OpenOffice.org 1.0, SOT Office or the upcoming StarOffice 6. You might be pleasently surprised.

Red Hat offers two embedded systems: Linux and eCos. I was wondering if the problem was more with eCos than Linux. But then again, I think the problem for Red Hat is that embedded is not their specialty, and if someone is going to go to a company at all for embedded Linux (they don't have to - after all) they will go to one that needs its embedded customers.

RedHat is a poor example because they're a late comer to the embedded world and only seemed to jump into it half way. Lynuxworks, maker of Bluecat Linux, recently laid off 30% [fuckedcompany.com] according to FuckedCompany.com. Lineo is basically done [slashdot.org]. How is MontaVista [mvista.com] doing these days? These are companies that are staking their whole business on embedding Linux.

However, just so people don't think it's just a Linux thing... Annasoft [annasoft.com], a leading Microsoft embedded partner, just died.

... because it was never as big and bloated as the Internet bubble. Furthermore, unlike all those Internet services and software packages that have just disappeared without a trace, open source software stays around: the investment in it isn't lost.

Of course, I do have my doubts that a company like Red Hat makes sense. Open source is best for specialized, highly-skilled consultants and professionals. It also makes sense for a few hardware vendors and large, established consulting shops like IBM. But companies like Red Hat and VA Linux aren't in that league.

I haven't got any spam trying to sell me embedded engineering manpower yet, like I got several for web design, Virusal Basic and other markets that have crashed recently.

I wonder, though, why companies aren't giving more money to the people whose software they use. Has Bruce Perens ever received a single penny for busybox? From any of the companies that use it? I don't know, but it certainly hasn't happened for me. It's sad that companies using open source software usually only spend money on training people who never did any embedded stuff rather than on the people whose ready-made software they are going to rip.

I wonder, though, why companies aren't giving more money to the people whose software they use.

Companies need long-term planning for their investments. Personally, I'd rather pay $1000 for software knowing that it goes to a company that is most likely going to be supporting the software for the next 2 years, than $500 for a package where this is not likely at all. Unfortunately, only very few O.S. projects come with some sort of established development process that guaratees that they're going to be "supported" (bug fixes...) in the next 2 years.

Also, the best Open Source programs are so popular and widespread that you get your support for them elsewhere (contracting...), so money spent by companies "on" these products goes to those other people. Finally, don't expect a company to donate money if it doesn't have some visible positive effect on the company itself...

IMHO, Open Source developers need to find business models that work well together with their products, e.g. web server software developers re-selling SSL certificates (I don't care where I get mine, and I don't mind paying slightly more if the money goes to the right people), compiler tools programmers offering affiliate links to recommended books about programming languages etc.... It's much easier to say "thank you" by chosing you as the reseller of something (not merchandise, unless it's something really useful!) you're going to buy anyway at more or less the same price, than by donating money with nothing in return.

Re-reading your comment, I noticed that you actually meant companies who included Open Source software in their commercial software packages... My rant wasn't specific to that situation, but to the use (not commercial re-distribution) of Open Source software by companies in general. So, it's just a rant;-)

eCOS, BSD, and RT Linux have established a clear and viable alternative to proprietary solutions, and they are expanding the extent to which embedded systems are becoming source-code compatible with one another.

For medium-to-large systems that could actually take advantage of 32-bit open source solutions, suddenly QNX, pSOS, and the others are talking price like never before. I'm sorry if Red Hat is losing money, but at least they've established a market that stops the established vendors from locking you in and gouging you.

Open source solutions have created an environment where the choice is no longer which vendor to be locked-in to because now there is the none-of-the-above choice. So now if you go with QNX because it is indeed the best option, they still have to be nice to you because you can walk. Everybody wins.

So the point is that the price of all embedded kernel source has equalized, proprietary or otherwise, the vendors now all compete on service, turn-key solutions, and technical merit.

You pay a one time fee for Tornado seats. What does it buy you? Some packaged up free software compilers (gcc!) and limited tech support. Same as Red Hat.

Outside of the support contract, you can get someone on-site from WindRiver for $2500 a day. And believe me, at some point, you will pay it because you've got to get your product shipping... sometimes you're willing to pay almost anything.

What makes them different? Per-copy license fees for VxWorks when you actually ship your product. And they ain't cheap. So guess what? There's a definite advantage to Open Source. No license fees! And the code works just as good as the closed source stuff, AND you don't have to beg WindRiver support for source code when things don't work right (AND this DOES happen). That's the customer perspective.

From Open Source perspective, one can make plenty of money just on support, just like WindRiver does. Adopt some open embedded code, learn it inside and out and start charging $100 an hour to support it. You can't help but be profitable as long as there are enough contracts around.

You'd be surprised at how many devices have embedded firmware.... toys, microwaves, cd players, radios, etc.,... ethernet palm pilot cradle I worked on a while back... barcode scanners... all manner of microcontrollers/firmware on computer peripherals (PCI cards, printers, scanners, etc.), cordless phones (do you think you get the caller ID message without firmware?) etc. etc.

And don't think for a minute that a single one of those devices runs anything that even remotely resembles a general-purpose operating system.