To judge by the vigorousness of the defense of its brand in recent weeks, it seems reasonably safe to conclude that open source can be evaluated as an institution in certain contexts. Granted, it’s a problematic approach generally given the fact that open source is not a single organization, but rather a loose federation of many individual institutions each with its own goals, principles and philosophies.

And yet for comparative purposes it occasionally is useful to look at the performance of open source as a whole relative to the commercial, closed source marketplace: itself a massive collection of independent entities, notably.

The occasion of OSCON is, in many respects, an entirely arbitrary endpoint for a check on the health of the phenomenon it is ostensibly organized around. At the same time, as Tim puts it, OSCON is a relatively unique gathering of the tribes, which perhaps makes such a system-wide evaluation entirely appropriate.

Depending on which particular business school text you pick up, you might have seen the organizational lifecycle stages described as some approximation of the following:

Startup

Growth

Maturity

Decline

We must of course acknowledge the glaring impedance mismatch between mixed motive movements such as open source and profit-centric enterprises. Undoubtedly, open source will occasionally, even frequently, follow a different trajectory than will closed source alternatives.

Still, if we may see differences between open and closed source businesses in how they experience those respective stages, it is nonetheless useful to ask the question: if open source did progress through a normal organizational lifecycle, where would it be today?

The LAMP Stack

With the term itself twelve years old come January, we can probably conclude that open source has passed successfully beyond the startup stage of its existence. The multi-billion dollar revenue stream generated by projects such as Linux validates this assumption.

If not startup, however, where is it? Growth? Maturity? Decline? It’s an interesting question. What’s more interesting is how you try to answer it.

Metrics in the open source space have always been problematic. Unlike paid software which can be assessed with moderate confidence via revenue metrics, it is inherently difficult if not impossible to track with any reasonable level of precision open source adoption, usage and so on. Proxies such as project download counts occasionally used by commercial entities or project backers are not entirely useless – particularly when you have a history – but they are woefully underrepresentative at best, fatally flawed at worst.

While we have our own means of tracking such things, we will occasionally turn to basic tools such as Google Trends for a sanity check.

As noisy as that is as a data source, it’s an interesting proxy itself for general traction. Search traffic, as a fundamental part of the experience for users and developer alike, tells us something about the activity around a given project. Provided, of course, there are no lurking variables from a query perspective; there’s a reason I never use Google Trends for LinkedIn’s Voldemort key value store, as but one example.

What can Google Trends tell us, then, about open source? Two weeks ago the folks from Black Duck kindly askedme to explore a related question: what does the future of open source look like? The obvious starting point for me was the venerable LAMP stack. As the most fundamentally popular combination of open source software, it seemed logical to look at the future of the space by looking at the relative performance of its most visible projects, as measured by Google’s multi-year dataset. The results were frankly surprising.

Here’s Linux.

Here’s Apache.

Here’s MySQL.

And here’s PHP.

Curious results, I think you’d agree. To assess whether or not this trending was unique to LAMP, I looked at “open source” as a query string.

And the GPL.

As an aside, just in case the Eclipse community is wondering why they weren’t polled, all I can say is that it’s your fault for naming yourself after a teen vampire movie.

Back to the graphs: it is certainly possible to argue that the project-specific results are an artifact of project-specific variables. The rise in popularity of distros such as Ubuntu, for instance, could certainly be involved in the decline in Linux-specific searches. But the fact that we’re seeing virtually the same trend line across all of the queried terms seems to indicate the presence of a larger pattern. The question is what that pattern means.

Curiously, Microsoft might hold some answers for us.

The Microsoft Connection

After a decade or so of stagnancy on the financial markets, we’re beginning to see more public criticism of Microsoft CEO Steve Ballmer. While there are a variety of arguments with his tenure ranging from the absurd to the substantive, the central point of contention in recent weeks has been the underperformance of the firm’s share price (see here or here). Both pieces use a graph created by Erik Pukinskis, which I don’t have permission to use and thus will not reproduce here. The basis for it, however, is the graph below. For context, Steve Ballmer was promoted to the Chief Executive position in January of 2000, according to Wikipedia.

Essentially, Ballmer’s tenure has been marked by a flat to slight decline in the company’s share price. Record results or no, Microsoft has yet to convince the street of its prospects for growth. The CEO’s defenders have hastened to point out that his ascension came during the height of the government’s anti-trust activities, arguing that fall out was inevitable, as was some sluggishness on the financial markets. But what the share price reflects, both to myself and another CEO I spoke with last week, is nothing more or less than a mature software business.

Ballmer once asked his own employees an interesting question: “we’ve had the two most successful software products in history: what makes you think there will be a third?” The lack of an obvious response to that question means that growth has been more difficult to come by, which in turn implies that Microsoft has transitioned from growth into a mature phase of the organizational lifecycle. Or at least that that’s the market perception: doubtless some at Microsoft would take exception to that idea.

Microsoft’s financial fortunes are not, of course, the focus of most of those within the larger open source community. But it is interesting, in spite of the different metrics, that the trendline of its share price – flat to declining – is not terribly dissimilar from the search trendlines we see for the various open source communities we’ve looked at.

The question, then, is this: assuming for the sake of argument that Microsoft is in a mature market stage, is open source as well?

The Growth

The short answer is that, at least by the very basic Google Trends metrics we’re using here, it is not. We know that usage of the vanilla Linux search term is declining. But what about Linux in conjunction with the cloud?

A healthy, growth-oriented trendline. And how about in the mobile space?

We can argue over how “open source” Android really is, but we can’t argue that it is based on Linux and that you can, in fact, get at least portions of the source. And again, that it is the opposite of flat-to-declining traffic. How about databases? Are they experiencing the same general abatement in interest that the longtime standard, MySQL, is above? Far from it.

We’ve writtenextensively about the explosion of interest in non-traditional (read: non-relational) datastores, and the Google Trends data reflects this. Growth for NoSQL is, effectively, growth for open source because virtually all of the significant projects in the space are open source. Mike Stonebraker, in his talk at GlueCon, stated definitively that you cannot be competitive as a new project in the database space without being open source.

Consider the trajectory of Hadoop.

Besides disproving vague assertions of some sort of open source malaise, Hadoop is evidence of the importance of open source. Would it be the growing standard for large dataset processing that it is if it was a closed source product? It’s difficult to say, but certainly there is no similarly popular commercial closed source startup.

The Net

While selective interpretation of data such as that collected from Google Trends has and will continue to fuel the kind of rhetoric that claims that open source is in its decline phase, the truth is that it is more likely to be early stage growth. That leaves considerable runway for those in and around the open source community moving forward.

It’s undeniably true that certain projects may have peaked from a relative popularity perspective: it’s difficult to see technologies like Java or MySQL (coverage) becoming more popular than they are today from a percentage of marketshare perspective. Increased competition and accelerating fragmentation are driving the market away from consolidated LAMP-style stacks towards more specialized, heterogeneous alternatives (coverage). But it is equally true that while the growth rate may be slowing for already ubiquitous technologies like Apache, MySQL and PHP, it’s accelerating for a variety of technologies in every area of the market from mobile to cloud. Where certain projects within the open source community are seeing slowing growth and a transition to more static market positions, there are a variety of other open source projects that are seeing dramatic spikes in traction.

Open source has been successful enough that it is essentially impossible for closed source infrastructure software startups to compete effectively, which is why we see fewer and fewer of them. Closed source software is not going away, clearly, and in addition to being an enterprise and consumer fixture it is still overwhelmingly the leading money earner by several orders of magnitude. But if open source has been this successful absent the types of revenue models we envision that more properly align its strengths with customer needs (coverage), what might we see in future?

The Outlook

More open source. Quite a lot more, probably.

Not strictly because of the inevitable innovation we’ll see in revenue models for commercial open source organizations. That will certainly add vital oxygen to certain open source ecosystems, but it won’t be a panacea. The business of selling open source software, remember, is dwarfed by the business of using open source software to produce and sell other services (coverage). And yet historically, most of the focus on open source software has accrued to those who sold it. Today, attention and traction is shifting to those who are not in the business of selling software, but rather share their assets via a variety of open source mechanisms.

Why would commercial organizations willingly cede the fruits of their labor to a market that might include their competitors? Because for software that is non-differentiating, that is not a competitive advantage – which for most non-technology firms is virtually all of their software – it will cost more over the longer term to author software privately than it would publicly. Facebook and Twitter demonstrate this quite adequately (coverage), true, but it’s not just the web firms. We see it when a hosting company (Rackspace) and space agency (NASA) jointly author a cloud computing stack that neither intends to create a software sales business around. We see it when Lockheed Martin launches an open source social networking project. And so on.

None of these can be characterized as decisions driven by idealism or emotion; they are simply the most logical means of developing software for companies that aren’t in the business of selling software. Make no mistake: we’re seeing a resurgence of roll your own software (coverage). The difference this time around is that by sharing the code developed internally as open source, it becomes possible to amortize the development costs across multiple organizations with similar needs. Worst case, you have the opportunity to lower your costs of talent acquisition; this, presumably, is one of the justifications for Google sharing details on its MapReduce and Pregel processing approaches.

The outlook for open source software, then, appears to be this: maturity in certain market segments, outpaced by growth overall.

Which is really excellent news, because like Tim, OSCON’s my favorite conference.

Disclosure: We have multiple clients in the Linux space, and we worked with MySQL prior to and subsequent to its acquisition by Sun, which was also a client. Also in the open source database family, Basho, author of Riak, and Cloudera, backer of Hadoop, are RedMonk clients. Rackspace, one of the authors of the OpenStack project, is a RedMonk client, as is Microsoft.

Do you happen to know definitively if google trends looks at terms relative to all searches — or relative to the term itself? For example, it is perfectly rational that PHP searches are down as a % of all searches — given how many more end users are online searching for information about vampires… I’ve never found a definitive answer for that.

As you said early in the piece, LAMP is nearly 12 years old and it is that, not that it is open-source, that is likely for the declining Google Trends charts and the characterization that it is in the maturity phase. Clearly many new open-source projects are growing as you’ve cited and so it is probably difficult to try to characterize all of open-source into one phase. Rather it is probably groups or categories of projects that could be characterized.

Interesting topic and discussion. To your point, much of the new mojo in open source is underground, powering some of the highest growth segments of the software industry – SaaS, Web 2.0, social networking. Average VC deal sizes are down dramatically, due in large part to the IP leverage and time-to-market advantages of open source scaffolding. We’re just scratching the surface.

What you’ve turned up with respect to the LAMP stack is the fact that the demographic of searchers in 2004 was more technical than in 2010.

I’d also argue that certain terms are searched more regularly when they are novel and when subject matter experts are providing information on the open web. Because of the professionalization of Linux, Apache, MySQL, and PHP, some searches may be conducted against vendor knowledge bases rather than the open web.

@Jeff: thanks for the clarification with respect to the calculation of the trends.

as far as demographics, i think it’s certainly true they’ve changed over six years, and i’m positive that the search destination for certain topics changes precisely as you outline.

all reasons to take the Trends data with a grain of salt, but i think you’re right in that they’re mitigating factors. the pattern is still interesting, i think, if only for its consistency across projects.

If you take a look at ubuntu as search term, you’ll get quite a different picture. It’s rising with a peak every half year. And that’s probably the main reason. It attracts new users, who need to google. And for each new version, people search for reviews, new features, different ways to change your settings.

If you have used apache for 10 years, you will hardly ever need to google for apache. You either use the pages you know or you search for something specific and don’t need apache a a search term.

This is a terrible way to judge whether something is “not startup.” You spent a lot of time on this. Too bad you couldn’t build anything reusable instead. Like code, you know? Or community? You are just making a mess in my opinion.

I don’t know; if I search for “software” in Google Trends I also see a clear trend downwards. I think it is just that some changes get ingrained into the discourse and become invisible. That is how success looks like, I guess. http://www.google.com/trends?q=software

Continuing the Discussion

[…] The State of Open Source: Startup, Growth, Maturity or Decline? Depending on which particular business school text you pick up, you might have seen the organizational lifecycle stages described as some approximation of the following: […]

[…] Stephen O’Grady has a great post analysing where the open source software movement is in the Startup, Growth, Maturity or Decline model. Why would commercial organizations willingly cede the fruits of their labor to a market that might include their competitors? Because for software that is non-differentiating, that is not a competitive advantage – which for most non-technology firms is virtually all of their software – it will cost more over the longer term to author software privately than it would publicly. Facebook and Twitter demonstrate this quite adequately (coverage), true, but it’s not just the web firms. We see it when a hosting company (Rackspace) and space agency (NASA) jointly author a cloud computing stack that neither intends to create a software sales business around. We see it when Lockheed Martin launches an open source social networking project. And so on. […]

[…] The State of Open Source: Startup, Growth, Maturity or Decline? O’Grady has come to the same conclusion that I did at OSCON, namely that the “open source bubble” may be over – the period where it was assumed open source would be directly monetised – and it’s time for a resurgence of the traditional approach of many parties with their core business elsewhere synchronising fragments of their interests with fragments of the interests of others. The announcement of OpenStack was iconic, but there were plenty of other signs once I’d started looking for them. […]

[…] been numerous reports about it, not least from the New York Times, but one that caught my eye was the meta-analysis from analyst Stephen O’Grady. O’Grady is characteristically detail-rich and his article is packed with Google Trends […]

[…] community and collaboration. He and Aslett refer to Stephen O’Grady’s post, “The State of Open Source: Startup, Growth, Maturity or Decline?“, which is rich with trend analysis over the years to show that some projects are in […]

[…] post begins with a reference to Simon’s post (and Stephen O’Grady’s The State of Open Source: Startup, Growth, Maturity or Decline? post which Simon also discussed) and comes to the conclusion that we are witnessing ”the […]

[…] Think big picture. Consider, for a moment, the component pieces of the venerable LAMP stack. Linux is the de facto alternative to Windows on the server and Apple in mobile, Apache is powering nearly sixty percent of the world’s websites, MySQL is the most popular relational database on the planet – important enough to hold up Oracle’s acquisition, and as for the dynamic languages, well, even Microsoft has been compelled to pimp them. Open source has so thoroughly transitioned to the mainstream, in fact, that some projects are exhibiting symptoms typical to mature businesses [coverage]. […]

[…] anti-competitive habits , either explicit or hidden that were common in past years. It also allows sharing of development costs across organizations. Younger generations are used to great user experience, and would not “go back” when […]