Posted
by
timothy
on Sunday December 05, 2010 @08:22AM
from the back-in-your-day-maybe dept.

Hugh Pickens writes "Believe it or not, there is still this illusion that Linux and open-source software is written by counter-culture, C++ programming cultists living in their parents' basements or huddled together in Cambridge, Mass. group-houses. Now CNet reports that the Linux Foundation has found that 'over 70% of all [Linux] kernel development is demonstrably done by developers who are being paid for their work.' That Linux is primarily developed by paid developers should come as no surprise considering that Linux enables many companies — hardware, software, and online services — to be more competitive in their markets and to find new ways to generate revenue. 'What's important about how Linux and open-source software is created isn't the side issues of politics or how its developers are perceived; it's that its fundamental methodology produces better software,' writes Stephen Vaughan-Nichols."

That is not part of the Linux kernel. It is part of the build system. As was rightly pointed out earlier by someone else bash, gcc, and the other build tools are not part of the linux kernel. You do not even need to use it to build the kernel, as you can simply make menuconfig rather than using the xconfig target.

I'm aware of that; just taking the opportunity to point out that C++ files on UNIX systems can use extensions other than those you mentioned, and in particular that the one bit of C++ in the Linux kernel repository uses the.cc extension. You also didn't cover the.C extension, though that one doesn't see much use due to incompatibility with case-insensitive systems.

"I'm aware of that; just taking the opportunity to point out that C++ files on UNIX systems can use extensions other than those you mentioned, and in particular that the one bit of C++ in the Linux kernel repository uses the.cc extension. "

Actually, you are only just now taking that opportunity. I appreciate the correction and additional information. In fact, that is exactly why I said I wanted to challenge the assertion, rather than saying there definitely isn't any.

I'm aware of that; just taking the opportunity to point out that C++ files on UNIX systems can use extensions other than those you mentioned, and in particular that the one bit of C++ in the Linux kernel repository uses the.cc extension.

Actually, you are only just now taking that opportunity. I appreciate the correction and additional information. In fact, that is exactly why I said I wanted to challenge the assertion, rather than saying there definitely isn't any.

The Linux kernel currently compiles with GCC, Clang/LLVM, PCC, Path64 (development branch only - some of the inline asm isn't handled correctly by the released version), ICC and (I think) XLC. An earlier version also compiled with TCC. It's not really that dependent on GNUisms. Or, rather, the GNUisms that it depends on are pretty well supported by other compilers.

Because Bjarne Stroustrup's C with Classes language is merely an early version of C++. I mean modern C has evolved quite a bit since that time, and it would be a shame to have to limit yourself to the C language constructs of 1983.

C++ is still the language invented by Bjarne Stroustrup, and he still writes the definitive book on the language. He is active in assisting in the development as it involves, and here is his FAQ on C++0x [att.com], which will be the next C++ standard expected to be released in 2011 according to his FAQ.

That being said, I believe the GP meant why not use C++, sticking to the subset of the C++ language that is very similar to C, but taking advantage of C++ classes. In reality, the only difference between a struct an

Sorry, I've written drivers, and etc, but never did any big kernel development. Yet, I don't have a clue why C++ (or better the subset of it that is C with classes - you can replace classes with C++ structs if you like, as you said, they are the same) is a bad laguage for kernel development. You don't gain only methods, you gain constructors and destructors whose execution are enforced by the compiler.

I didn't yet see where control is lost. For me, it seems to be the other way around, the compiler will give

"you gain constructors and destructors whose execution are enforced by the compiler.

You pretty much answered your own question without knowing it. Overhead enforced by the compiler is a BadThing(tm) when trying to create an efficient OS. With OS development you are doing lots of data structure related access, and you don't want indirection except when absolutely necessary, for example.Here is Linus explaining the issue quite well (from a Linux Gazette article [tldp.org]):

Well, that actually isn't an English sentence that parses. Obviously, one or more elements in a C struct can be pointers to functions, as anyone who has even a passing familiarity with the Linuk kernel code knows. This is completely different from a C++ member function, however. The part about memory corruption detection doesn't really make any sense. Can you give another try at expressing what you were trying to communicate to us?

Many fields in a struct will not immediately cause the execution of a process to stop. For example you can corrupt an int and the program may keep running. Corrupt a float and you may or may not get a floating point exception but corrupt and then call a function pointer and its all over.

"Corrupt a float and you may or may not get a floating point exception but corrupt and then call a function pointer and its all over.

Not necessarily. It depends on how it gets corrupted. If the corrupted space turns out to translate to an address that points to code it may well cause behavior that may be either be subtly bizarre or completely innocuous.

The Linux kernel (which isn't explicitly mentioned in the summary) is written in C, as are numerous other Open Source packages that are used in Linux distributions. A purist might say that only the kernel is Linux, but we don't usually talk about GNU/Linux distributions, even here, so expecting others to make the distinction would be patently absurd. KDE is written in C++, as is a lot of other Open Source software including QT. It is also true that a lot of code is in Perl, BASH, SED, AWK, Python, Java,

The Linux kernel is written in C, not C++. And haven't there been a number of articles on how IBM, RedHat, Sun etc all have employees who develop Linux?
One post in particular... [slashdot.org] and I know there have been others.

My employer uses F/OSS extensively - and as the sysadmin, I've started to notice a pattern.

F/OSS products which scratch an individual or a small group of peoples' itch generally get developed to a certain point and then stagnate. If you're lucky, that point is acceptable to you.

The products that do really well - the "best of F/OSS", if you like - are almost invariably the sort which scratches a very common itch. They're usually bankrolled by a number of companies (the Linux kernel falls under this category) or become self-funding when the project leader sets up a company to sell a commercial version with support and possibly extra features.

One real benefit is that if you are a company developing kernel code and contribute it back you will get goodwill and you will enhance the competence of your employees.

And the Linux kernel is a known stable solution that will be around for a while. It is also easy to find people with knowledge about the kernel itself (to some extent at least) which makes it easier for a company to find competence instead of having to waste training on the employees and hope that they will pay off in 6 to 12 months.

One real benefit is that if you are a company developing kernel code and contribute it back you will get goodwill and you will enhance the competence of your employees.

Actually, I've long thought it strange that the business/industrial world has an objection to supporting things like an OS kernel, runtime libraries, etc. The obvious parallel comes to mind: Lost of companies farm out part of their operations to subcontractors. They routinely subcontract for cleaning, delivery, electrical services, for example, not to mention their phone, water and sewage systems. They don't seem to be taken aback by the fact that the companies that supply these services also subcontract to "the competition".

The idea of paying a separate company for software development and supports services is also hardly new. That is how IBM has made much of its money, after all. Paying a company like Red Hat doesn't strike me as very different from any of the above. It doesn't take much management genius to understand that paying a contracting firm for software support at the "system" level is a fairly good idea. That way, you can share the cost with all the other companies that hire the same software firm, and everyone can get the benefits from having the software organized by people who (hopefully;-) know what they're doing.

So why is this even a story? You'd think there would have been enough sensible businessmen all along for lots of Red Hats to prosper.

A related question is all the propaganda against "open" software. Systems such as water, sewage, electrical, etc. all have "open" designs, with everything published and the detailed specs easily available to anyone. Companies don't often buy fleet vehicles without shop manuals, which give the detailed specs for the innards of the vehicles. Why would people classify open software as "hippie" or "communist", when they don't say the same about shop manuals or electrical diagrams? You'd think that sensible managers who approve of open standards for these other things would also want software that follows published standards (e.g., POSIX), and whose specs (i.e., the source code) is easily available to everyone.

But for some unexplained reason, business people keep buying software systems with hidden, "proprietary" innards. They wouldn't do this with delivery vans or electrical wiring; why would they accept it with software? Exactly the same reasoning says that software should be open, standardized, and accessible to anyone with the technical training. And the same reasoning that supports specialized firms to do common tasks should also support specialized firms for software needs.

It may be yet another example of a theory that keeps popping up: Whenever a computer is introduced anywhere, all precedent is forgotten, and people have to relearn from scratch all the things that they knew from before there were computers. I wonder what it is about computers that causes this social amnesia and inability to see parallel situations?

Well the same is true anywhere, the more potential users a given piece of software has the more interest there is in developing it...

Commercial software works the same way, something with mass market appeal is highly likely to be developed and either result in multiple competitors or serious effort to stifle competition...But something small with a very limited market either won't exist at all, or will be extremely expensive if that niche market has the money to pay for it. Niche products also tend to be rather buggy.

OSS serves certain niches very well, ie those niches occupied by technically minded people who are capable of writing what they need for themselves... Other niches are served somewhat less well because those who would be capable of producing software have no need for it, and those who need it aren't capable of writing it.

You also get a lot of businesses and individuals using software which is extremely poorly suited to their needs because they aren't able to customise it for themselves, and the only people who would be capable of doing so aren't willing to.

I'm sure there are many things that could be improved by being computerized, but where the people capable of producing such software either don't realise or don't care.

Other niches are served somewhat less well because those who would be capable of producing software have no need for it, and those who need it aren't capable of writing it.

The solution is rather simple. If you know enough people who need a particular software but can't make the software themselves (artists, musicians, authors, etc) make a donation pool. The first developer who will create the software will get the donation pool but the software is free (both open source and free of charge). The cost for each client are minimal (like 10$) but the pay for the developer will be good (if 1000 people donate each 10$ that's 10000$ for the developer). The end product is a free softw

Or expensive. Some niches are defined by having extremely high quality requirements on entry, which make it hard for non-commercial software to get started precisely because none of the users will consider any option that isn't very good indeed. Those tend to be pretty conservative niches, but for some things (e.g., anything safety-critical) that's good; it's reasonable to spend on having things dependable.

F/OSS products which scratch an individual or a small group of peoples' itch generally get developed to a certain point and then stagnate.

Indeed. I use it extensively, and have noticed that they always forget to put in the malware, nagware, and crapware parts.

I suppose you could adopt a business model whereby you'd put those things in there and charge a premium just like closed-source software, but since you'd have to make the source available somebody could just take it out again. Why anyone would want to, I don't know.

The products that do really well - the "best of F/OSS", if you like - are almost invariably the sort which scratches a very common itch. They're usually bankrolled by a number of companies (the Linux kernel falls under this category) or become self-funding when the project leader sets up a company to sell a commercial version with support and possibly extra features

Interesting quote.

It seems to work almost the way Richard Stallman would have hoped. Somebody likes a piece of software enough to pay a program

The thing is, those other people have to make any of their own improvements available. Compare and contrast that with the BSDs, where there is no obligation to re-release improvements. Lots of expensive, specialist kit is based around a BSD Unix (eg. F5, Juniper). But BSD doesn't have anything like the mindshare in the generic server market.

This is incorrect on two levels. First, you only need to make the changes available if you distribute the changed code to outside parties. Second, you only have to make the changes to those outside parties, not the general public. However, those parties are free to redistribute it, so this really depends on your customers.At my company, we have modified several GNU tools, and we haven't released any source code because we use them internally.

I don't see how money is being made with "several companies bankrolling" a piece of software that scratches a common itch. Is it that a particular company sees that it can make money by using, though not owning, a particular piece of software and they don't mind their money making improvements available to other people?

Typically it's "bankrolling" by assigning some of your people to spend some of their time on making improvements you happen to need and contributing them upstream.

(Contributing improvements upstream means that you won't need to continually maintain your patches, that they'll eventually be included in vendor packages/kernels and thus that you'll later need to do less packaging yourself, and is otherwise a money-saving action. As a somewhat-related aside -- once upon a time I worked in embedded Linux, and you had companies who structured their default contracts for kernel work such that everything was submitted upstream, making each contract effectively a one-time engagement when everything was done right, and others who didn't... ehh... encourage their clients to pursue submitting their code, such that said clients would keep paying in to keep their patch current with newer upstream kernels).

Well, more people = more developers. But when it comes to paid developers, I'm not sure the same is true. If it becomes too common an itch that you're trying to collect microfunding from millions of people then it's not doing that great. Even though millions of people need a little photo editing they aren't funding GIMP and the professionals rather end up paying for Photoshop, same goes for OpenOffice vs MS Office. Firefox is a bit different in that Google is paying indirectly rather than the users directly

Gimp isn't getting much funding at all. The glacial pace of development demonstrates that pretty well, IMV.

This demonstrates another important point about open source: it is more important that development should never stop than that it should proceed at some particular pace. With all its flaws, Girmp is still a powerful tool that gets the job done. As it happens, I just finished an hour with Gimp doing some pretty sophisticated image manipulation. An example of one WTF moment: rectangular select comes complete with a dialog box in which exact pixel region dimensions can be entered, exactly what I need. You

gimp had the issue that none of the developers thought the UI was broken, and iirc they have been quite happy to bring interface customisation code into the main line if anyone could be arsed to write it (and, i assume, it doesn't suck).

In short, information isn't ever really free. Someone, somewhere is either donating their valuable time to development or there's an organization funding the devs. Nobody can really afford to slave away coding without compensation - everybody has to eat. The for-profit model works because it's a fair exchange of value for both parties.

This might seem like an unnecessarily elementary point, but there seems to be an awful lot of people who sincerely believe in the adolescent idea that work gets done by mag

Believe it or not, there is still this illusion that Linux and open-source software is written by counter-culture, C++ programming cultists living in their parents' basements or huddled together in Cambridge, Mass. group-houses. [...]That Linux is primarily developed by paid developers should come as no surprise

Hang on a minute. If there's an illusion that it's written by smelly vegan hippies, then surely it should come as a surprise that it's written by paid employees of €eevu£ corpra$hunz.

Though the question is actually irrelevant - everybody has known it for years. It's a non-story.

This perception of OSS still remains rather strong. You will notice that a common advocacy for OSS is the "many eyes" thing. The idea seems to be that there are just tons and tons of developers out there with amazing amounts of time that will jump on a project and help, if only it was opened up. The advocates then point to things like Linux or Firefox or MYSQL and how great they are. What they miss or don't understand is that these high profile, top-flight OSS projects have heavy financial backing. They have developers who's full time jobs it is to work on the software. That's wonderful and all, but don't then try to claim that you'll get that kind of development on a project just because you open up the code.

Actually, most of the people who miss that are not listening closely enough. The point of the "many eyes" thing with Open Source software is that if an OSS project is of interest to a company (in particular a company that hires programmers anyway) that company can have its people look at the software from the perspective of the way that company uses the software and they can make changes (either bug fixes or otherwise) to make the software better for their company. If they were using proprietary software, they would be at the mercy of the company that owned the rights to that software for any fixes or improvements.

If they were using proprietary software, they would be at the mercy of the company that owned the rights to that software for any fixes or improvements.

If they're interested in that, which IMHO they are mostly not. Many companies have learned the hard way the true cost of custom software, which in many ways can be worse than proprietary software. If there's at least a semi-functioning marketplace, the proprietary companies have to evolve their solutions and keep their prices (thus your costs) to what the market will bear. Custom software OTOH stands dead still unless you pay every inch of the way, which is why they're not interested in doing anything custo

If they aren't interested in that, how do you explain all of the companies that pay programmers to work on Open Source?
Most (if not all) of those companies already had programmers on staff to write custom software for the company. They discovered that it was easier and cheaper (and often better) to modify Open Source Software than it was to write their own application from scratch (or to buy such an application from some proprietary vendor).

Many companies have learned the hard way the true cost of custom software, which in many ways can be worse than proprietary software.

If you're making custom software out of your OSS software, you're doing it wrong.

If you're doing it right, you're submitting your changes back upstream -- so the software doesn't "stand dead-still", as you put it, even on those times when you aren't shoveling man-hours into improvements. If you happen to be curious for some examples, google around for patches under my name submitted to open source projects over the last decade. Just about all of those were paid for by my employers -- from the OS X VNC plugin bugfix to the feature enhancements to libvirt to improved cover page generation for HylaFAX.

For the work I did at Dell, we worked together with Red Hat to get as many of the libvirt and qemu improvements we wanted as possible into the RHEL6 release schedule, enabling some of Dell's internal QA tools to work out-of-the-box with RHEL6 (whereas those same tools required heavy tweaking on RHEL5). Sure, we could have gotten the same thing done as a professional services engagement rather than a friendly collaboration between engineering groups... but this way was far easier, cheaper and lower-paperwork (and by building the patches in-house, we made sure that we got exactly what we wanted).

What if you can't get it, because it conflicts with the interests, goals or methods of the developers of the package you want to customize ? A good example for this case is the android wake-lock, which was rejected by the kernel team, and one year after there was still debate about how to replicate the function. During all that time, there were companies shipping products with the Google patch, but it was not integrated into the original kernel tree.

What if you can't get it, because it conflicts with the interests, goals or methods of the developers of the package you want to customize?

Then you've got a decision to make, obviously -- comparing the costs of maintaining your patch in-house versus the benefit it provides.

These are fairly uncommon edge cases, however. Upstream may fairly often reject a specific implementation, but it's generally quite possible to work with them to come up with a solution for your problem which makes everyone happy.

If you're making custom software out of your OSS software, you're doing it wrong.

Not at all.

The freedom to run the program, for any purpose is Freedom 0.
The freedom to study how the program works, and change it to make it do what you wish is Freedom 1.
The freedom to distribute copies of your modified versions to others is Freedom 3.

If you are obligated to redistribute your modified version, then it's not a freedom at all, because you have no choice.

There's no legal question involved, as the "Four Essential Freedoms" are part of the GNU philosophy, not part of the licensing. They are principles behind "Free Software." If you want to argue that OSS is different from Free Software, that would be fine. Just specify whether it differs in what freedoms it protects or what obligations it imposes.

So, no misunderstanding at all - I am saying that it's NOT wrong to modify and use OSS solely for your own purposes.

Of course opening the code doesn't make it automatic. But closing it often precludes such change or sophistication: when I have professional access to the software base for commercial packages, I'm often _amazed_ at the boneheaded practices I'd pull a release candidate for on the spot, and make the author go back and rewrite it during our code review meetings.

The idea seems to be that there are just tons and tons of developers out there with amazing amounts of time that will jump on a project and help, if only it was opened up

This is a common misconception about free software.

"Many eyes" does not mean every user is a developer, as a matter of fact the vast majority is not.

What "many eyes" means is that IF a user is bothered enough about a bug and that user has the ability to develop software, then he CAN fix that bug. There might be a million users, but if only 0.1% of them are interested developers then there will be a thousand people fixing that bug.

Actually, GNU HURD seems to be mainly written by academics. It suffers the fate of most similar projects - lots of technically very interesting work is going on, but there's no strong motivation to bring it to a state where it can be used by the general public.

No, what’s important about how Linux and open-source software is created isn’t the side issues of politics or how its developers are perceived; it’s that its fundamental methodology produces better software. That’s why businesses invest in Linux’s development. Linux works. If it didn’t, big business wouldn’t bother with it.

I am not sure the methodology produces better software; it does however produce software companies can use and modify to their needs without paying ongoing license fees. That's why they use it - it allows them to develop other, higher-value, products and maintain control over the source and not be beholden to some third party company. The methodology does work to produce high quality software since many people are looking for bugs, with that part I agree - but companies don't invest in software because others are QC'ing it; they invest because it enables them to make money. While what the author says is, IMHO, true about the Linux kernal, I don't think it is true for many other FOSS projects; unless they are used as part of a larger product, such as a server. There simply isn't the same incentive to spend time and money on an application, such as an office suite, that you can't use to sell something more profitable.

I am not sure the methodology produces better software; it does however produce software companies can use and modify to their needs without paying ongoing license fees. That's why they use it - it allows them to develop other, higher-value, products and maintain control over the source and not be beholden to some third party company.

I think there's another piece or two as well, and I think the bit about "no license fees" is overstated. Sure, not having license fees to pay is nice, but in most cases companies who are building profitable products wouldn't care about paying some money for an OS license. They can buy something like QNX for relatively small amounts of money and customize it however they like, without the requirement that they publish their changes to potential competitors. So why don't they do that?

I doubt this has been the case for 10,15 years or more. The fact is that most of these "counter culture" people are inventions of the media or hollywood and have never, really, existed in the real world. The few people who would describe themselves as such may still exist in some parts of the world, and are usually referred to as criminal hacking gangs, but they've not contributed a dam' thing to the Linux kernel.

That's not to say some individuals with long hair and others with low personal hygiene standards haven't done their bit, but those attributes don't make you counter-culture.

It has seemed to me for a while that the really popular pieces of FOSS (or the majority) are done by professionals. The bearded, Simpson's comic book store owner types come in afterwards either to complain about how something isn't politically correct enough to get people to make a castrated, completely free version of it. That or they pop up on Slahsdot talking like they are the ones who built the stuff and dissing people who want the software to be friendly.

The GPL facilitates (forces) the work to be shared among stakeholders. My company buys maintenance and feature development on Postgresql just the same and it's a really great deal for all of us who contribute to eachother not having to buy Oracle licenses.

"The Linux Foundation's report (PDF) on who writes Linux — "... Linux isn't written by lonely nerds hiding out in their parents' basements. It's written by people working for major companies — many of them businesses that you probably don't associate with Linux. To be exact, while 18.2% of Linux is written by people who aren't working for a company, and 7.6% is created by programmers who don't give a company affiliation, everything else is written by someone who's getting paid to create Linux. From top to bottom, of the companies that have contributed more than 1% of the current Linux kernel, the list looks like this:..."

I've been working as an embedded Linux developer for almost a decade now, and yes, most of us kernel hackers are paid for our work. For example, right now I'm working for a major microchip manufacturer that wants to make sure their products are fully supported by Linux. Consequently, they fund teams of open source developers (often hired through big name consulting firms) to port the kernel to their latest CPU's, develop drivers for integrated peripherals, etc. Just look at the email addresses in the submit logs for major open source projects. You will see ibm.com, intel.com, ti.com, redhat.com, windriver.com..., and many, many more big commercial technology companies. Its been this way for a while, which is why I would always laugh whenever some MS fanboy would try to denigrate Linux programmers as a bunch of basement dwellers. I make a better than average living from Linux coding, with multiple job offers right now, even in this horrid economy.

This is also why I have no worries about Microsoft ever killing off Linux. There are far too many companies making far too much money from Linux based products in market niches that MS has no traction in. The embedded and mobile markets are pretty much owned by Linux, and those are pretty much the only tech sectors seeing strong growth right now. If you haven't yet added Linux skills to your resume, do it.

If anyone wants to ask me about the Linux / embedded / open source consulting world, go ahead and post your questions. I'll check back and answer if I can.

Embedded programming requires a certain mind set. Board support packages, memory organization, interrupt handling schemes, latency management, working around chip set bugs/oddities, reverse engineering, close collaboration with hardware designers for field programmable gate arrays or ASICs, bootstrapping, and system integration are NOT taught in any school I have discovered. The closest I have found is Computer Engineering programs that offer robotics or some other courses/projects that include interfaces

Fandom House was in Somerville, blocks away from Cambridge. And to be clear, most of the people living there a) had worked at BBN at some point and b) earned over $100K/yr in current dollars. (The house on Linnean St. was in Cambridge, true, but I might point out that the basement their was converted into a sauna level with five cold and hot pools, hardly what the OP implies.)

As one of those 'sell outs', I'm curious why you think that? At the heart of the open source ethos is the license under which the software is distributed. As long as the code that is developed is submitted back to the open source community, that ethos is satisfied. How or even if the developer is paid is not really relevant. Indeed, a world in which most commercial software is open source and the developers are paid for their efforts is very much in keeping with our dreams of an open source utopia.

(...) a world in which most commercial software is open source and the developers are paid for their efforts is very much in keeping with our dreams of an open source utopia.

Indeed. The most paid F/OSS developers there are, the best for everyone. If there is demand, F/OSS developers salaries will raise, which will make more people interested in working in the field. The result will be better software for everyone.

As one of those 'sell outs', I'm curious why you think that? At the heart of the open source ethos is the license under which the software is distributed. As long as the code that is developed is submitted back to the open source community, that ethos is satisfied. How or even if the developer is paid is not really relevant. Indeed, a world in which most commercial software is open source and the developers are paid for their efforts is very much in keeping with our dreams of an open source utopia.

YHBT. YHL. HAND.

I would think that with a five digit UID and a paycheck from hacking the kernel, you would be able to recognize this.

I, personally, have no problem with these developers being paid. Open source is not always free and vice versa. I think what's important is the code remaining accessible.

Kind of hard to do that and follow these rules. Personally, I think Open Source licenses are crap. You give your work away for free, but acknowledge it has some value at the same time (by preventing others from making private contributions and commercial redistributions). Which is it, really free, or not? If you have confidence in yourself, sell your work. It leads to higher quality than "here is some free crap, don't touch it without giving me all your work for free too". Like a low quality guarantee