Category: Open Source

I am currently messing around in the pits of .NET e-commerce. I thought it would be the last place I’d find open source inspired disharmony. But no, even here it is to be found. 😉

OK, a bit of background.

NOP Commerce is an ecommerce platform based on Microsoft’s open source ASP.NET platform. The project has been around for five or six years or so. Gets very good reviews too. Last year SmartStore.NET forked NOP. Nothing wrong with that, NOP is GPL’ed. That would be fine except for a clause in NOPs license which states that you must keep a link in your website footer to the project website unless you pay a small $60 waiver fee.

The problem, and the tension, comes from SmartStore.NET having removed the link requirement from their fork.

Whatever the legalities involved, and I am not legally qualified to comment either way, the SmartStore.NET fork doesn’t feel right. The NOP guys have put a ton of work into the project and they deserve better.

The sad thing is that there is a lot in SmartStore.NET to like. Wouldn’t a better option have been to merge the changes into NOP Commerce so that everybody wins?

Update: if you are after a .NET based e-commerce system then Virto Commerce is worth a look. Looks to be maturing qickly.

OpenNMS – whilst not a dedicated event correlation tool, OpenNMS does contain an event correlation engine based upon the Drools engine mentioned above.

Esper (and Nesper) – Esper is a Java based components (Nesper is a .NET based version of Esper) for complex event processing.

If you want a survey of event correlation techniques and tools, you could do a lot worse than read Andreas Müller’s master’s thesis titled Event Correlation Engine. It is a few years old, but is still pretty current.

NetFlow is a standard from Cisco for transferring of network analysis data across a network. The last thing you want to do with your routers and switches is give them the burden of analyzing network traffic, so Cisco came up with NetFlow so that you can offload the analysis to less CPU bound devices.

NTop – a traffic analyser that runs on most UNIX variants and Microsoft Windows. In addition, ntop includes Cisco NetFlow and sFlow support. For an introduction to NTop, please see this introduction to NTop video.

Flow-tools – a library and a collection of programs used to collect, send, process, and generate reports from NetFlow data.

A bit of background

Zenoss is written in the Python programming language and uses the Zope web application framework.

Relstorage is a Zope add-on for saving data to a relational database. Relstorage allows Zenoss to use a relational database as its storage backend, with all of the scaling out benefits that entails.

A relational database does not need to run on the same server as Zope, you can run it on a completely different server. In fact, you can run the relational database on a cluster of machines giving substantial scalability benefits. With Zope’s native database format, running it on a different machine isn’t possible.

Zope’s native database format limits how scalable Zope can be, which in turn limits how scalable Zenoss can be.

“a mistake we made in the process was that product management had it slated for an enterprise feature but we began development in the core trunk. for better or worse, our decision was to pull it back. we are continuing to look this as part of the plan forward.”

“all good points. to be clear, the Core/Enterprise feature decisions are challenging and involve tough trade offs.”

As Bill Karpovitch says above, the relstorage feature wasn’t intended to go into Core, it was supposed to be an Enterprise only feature.

Unfortunately, it kind of snuck into Core accidentally and was removed. This upset the Zenoss community quite a lot as they believed that it would effectively create a fork between the Core and Enterprise products. A fork would make creating add-ons more difficult because potentially Core and Enterprise would need to be tested seperately. Supporting both would also be more difficult due to the core of the Enterprise product being different from Zenoss Core.

Conclusion

BTW this post is in no way an attack on Zenoss, or their very fine business. Kudos to them for having this kind of discussion in public. The above is just a concrete example of the friction that is inevitable when you have an open source product and a closed source product.

The friction isn’t confined to open core companies either, closed source companies have exactly the same friction when they have tiered products too.

It has been six years since I wrote about Network management’s “new wave” and thought it would be interesting to go back and see what has happened. We are now at the outer envelope of the VC funding cycle so things should be sorting themselves out one way or another.

Are they still open

Looks like Groundwork isn’t that open. Groundwork Monitor Core product is restricted to 50 devices. The license doesn’t look at all open.

The open source moniker has gone. Hard to tell that, say Zenoss for instance, is actually open source by looking at their home page. If you were an alien just off the mother ship and only had the home page to look at, you wouldn’t know that the core product is open source.

Effects upon closed source competitors

I suggested that the “new wave” would have the effect of opening up the “big 4”. I can see no evidence of this at all. I also thought that the “big 4” would be good candidates to buy the “new wave” and that hasn’t happened either.

Effects upon consumers

The one big winner has been users. Open source network management software ten years ago could be hard work with no proper packaging and woeful documentation. Now, there are some really nice options that are much easier to work with. There are also large communities as well to offer support and guidance where necessary.

Conclusion

I find it hard to believe that too many people would consider the “new wave” experiment to be a major success story. I’m not saying it has failed, but venture capitalists invest money to win big, and the investment hasn’t won big. It probably didn’t help that the financial meltdown happened.

There are a number of winners, not least among the many users who have high quality software to use at minimal cost.

I doubt that venture capitalists will be rushing to find their cheque books to fund another round of open source network management companies.

Had the same wave of money been invested in closed source companies doing the same thing I’d bet that they would have been more successful in strictly money terms at least. If a user jumps on board your ecosystem for the sole reason they can get your core offering for free, is that user going to be worth the same in the long term as a customer who literally bought into the ecosystem? My expectation is that they are not.

I am not saying that open source businesses aren’t perfectly viable businesses. It just means they may not be as profitable as an equivalent closed source business. And money in the end of the day is what venture capitalists are interested in.

A functionality ceiling is a level of functionality beyond which the community edition product manager is unwilling to implement because of the fear that the enterprise product will be less attractive to potential customers.

That got me thinking, if a functionality ceiling does exist, how can I demonstrate it?

The graphs below are taken from the Ohloh open source project directory. The rather useful thing about Ohloh is, in addition to cataloguing open source projects, it also performs extensive code analysis.

Both of the graphs clearly show a plateau in the quantity of code committed to the respective community edition code repositories. There may be a number of explanations for the plateau, perhaps heavy re-factoring work clears the space required by new features. Though, realistically I doubt that re-factoring would be capable of continually reducing the size of the code base in order to make way for new code.

The plateau look suspiciously like evidence that open core software, at least in the network management world, tends towards a functional ceiling.

Open core refers to a business strategy employed by some commercial open source companies. The open core strategy is popular amongst companies within network management.

The open core strategy is largely defined by creating an open source community product that is freely given away, and another product, the enterprise edition, that is sold as a regular commercial software product.

The open core business model is useful to software vendors because it permits them to build a community surrounding the open product who will form the nucleus of the people who upgrade to the enterprise product.

The enterprise product is useful because it is packaged and sold in the same way as proprietary software. One of the major pluses for the open core strategy is that, having a paid for a product with all of the sales infrastructure that implies, fits in with many company’s purchasing processes. Tarus Balog, project lead of OpenNMS, posted about how his pure play open source business sometimes struggles with companies who expect to purchase software, rather than deploy the software for free and pay for training and implementation services.

The shareware product is usually upgradeable to the full version by entering a product key supplied when the full version is purchased.

Whilst there is at least a grain of truth in the comparison there are some key differences between shareware and open core:

Key features are missing – open core software is useful in and of itself. An open core product that isn’t functional will not gain traction with a user community;

No community contributions – open core companies are keen to develop a community around their open core offering and hope/expect the community to contribute to the software eco-system surrounding the open core project;

Time limited – open core software is not time limited, you can use it for as long as you want.

The main similarity between open core and shareware business models is the desired end result on the part of the publisher. Both business models hope to up sell users to the full version of the product. The method used by both is also very similar, both business models withhold valuable functionality until the user upgrades.

In many ways network management is a perfect environment in which to exploit an open core strategy. Network management products are commonly structured around a central engine with add-ons integrating with third party networking hardware and servers.

The enterprise product is built around the core engine with a number of add-ons not provided in the community edition. The dual product strategy is most clearly taken by Zenoss who provide an open core engine but withhold many useful add-ons for important enterprise services like Microsoft Exchange and Active Directory. Whilst anybody could use the core engine to write their own add-on to provide the same functionality, many organisations find it more efficient to pay for a ready made and tested solution.

Vendor perspective

The pros and cons of the open core business model from the vendor’s perspective.

Open core licensing

The central part of an open core strategy is the dual licence. The community edition product is licensed under an open source licence, the enterprise product is usually licensed under a proprietary licence. Sometimes, when copyright or licensing issues intrude, the enterprise product also has an open source licence. Groundwork Monitor Enterprise Edition is a good example of an enterprise product having an open source licence. Dual licensing is only possible if you hold the copyright to all of the code, or have the agreement of the third party copyright holders to distribute under a restrictive licence. The same applies to any libraries distributed with the enterprise product.

If the enterprise product is licensed under an open source licence then there is always the danger that a customer may release the product in public, including the source code, meaning that potential customers no longer need to purchase the enterprise product in order to get hold of the value added features.

A fork in the road

A rival copy of an open source project based upon the same source code is called a fork.

As the core community product is freely available to anybody, there is a danger that a third party could create add-ons to the community edition and sell them in direct competition to the open core company. Whilst there is a danger of a competitor emerging to utilise the community product, there are some very good reasons why it won’t happen.

The competitor would be barred from using trademarks from the community edition in their product name or website. Consequently, it would be very difficult to promote the add-on to the desired audience. Trademark issues were one of the causes of the Icinga Nagios fork for instance.

In order to get around the trademark issue, the competitor would be forced to fork the community edition and release it under a new name. They could then sell an add-on product. Plainly the original community wouldn’t know anything about the fork and it would take a lot of marketing effort, in an already competitive market, for anybody to notice.

With the original community largely closed off, the competitor would have to start afresh and build a new community from scratch. Building a community takes time and money, external investment would be a very useful way to kick start the process. The competitor would not make a particularly attractive target for investment given that it doesn’t own any of the intellectual property of the fork.

In addition, the competitor would need to be absolutely certain that there are no source code or other artefacts which are being distributed as an exception to the community edition licence. There may also be clauses in the licence that have been inserted to guard against forking. The Zenoss licence contains just such a poisoning clause for instance.

Whilst forking is a danger to any commercial open core company, it does not appear to be a very pressing danger in practice.

At the other end of the spectrum, Groundwork Open Source have executed more of an aggregation strategy, by bundling together well known open source projects together and making them into an enterprise network management platform with their own glue software.

The aggregation strategy could be considered to be more in keeping with the open source philosophy of software reuse. There are a number of advantages and disadvantages to the aggregation strategy. The main advantage being that by reusing a lot of best of breed components you get to market much faster than starting off from scratch. Though the vast array of licenses used by the various open source projects are likely to keep a good number of lawyers busy trying to sort out all of the requirements. Some open source licenses may well preclude use of the software within a commercial setting. Porting the software to new platforms is also likely to be difficult, you can only support the union of all of the platforms supported by the projects. Without the agreement of the project leads, you may have problems with trademark use, especially if you wish to market your software as being powered by the project in question. Many open source projects, Nagios being a very good example, do protect their names quite vigorously.

On the positive side, if you can leverage existing projects, you will have a number of communities ready and waiting to be up sold to your enterprise product.

Community perspective

An exploration of open core from the community perspective.

The Open Core Functionality Ceiling

Does having an open core strategy mean that there are features that will never appear in the community product? Does the requirement to provide sufficient leverage to the sales VP provide an artificial ceiling for the functionality of the community product?

In a fully functioning open source eco system, the community would tend to close the gap between the community product and the enterprise product. Plainly an open core company is not going to be very comfortable with the value proposition of the enterprise product being undermined by the community.

Community contributions in an open core world

One of the problems with the open core strategy from the vendor perspective is that you need to be careful with how you handle community contributions. In the case where the company takes no third party contributions this isn’t going to be a problem, all of the engineers are on the company payroll.

If the company accepts third party contributions things become quite complex. In order to create a proprietary version of the software you either need to own the copyright to all of the software or have some kind of agreement to use the software in that way. A good example of such a third party contribution agreement is the Rivermuse contribution agreement.

The Rivermuse agreement must be signed each time a contribution is made. Whilst, from Rivermuse’s perspective, the agreement is absolutely necessary, I would think that the terms might make third party contributors think twice before agreeing to it.

Not only do Rivermuse have the right to sell your software without compensation, you have to assign the copyright of your work to Rivermuse. They also have the right to apply for patents based upon your work. If you submit your code, you could find yourself being sued for patent infringement by Rivermuse for discoveries that you made in the first place.

The OpenNMS Project Contributor Agreement, like the Rivermuse contribution agreement, also mandates that contributors assign copyright to the OpenNMS Group. The major difference is that the contributions are effectively owned by two parties, the contributor themselves and the OpenNMS Group, an arrangement known as dual ownership. The contributor also grants the OpenNMS Group a licence to use any patents contained within the contribution.

Open source etiquette

Many open source projects are written by people who gain no financial benefit from doing so. Open source software has been around for long enough that certain modes of behaviour have become the norm. One of the norms is the expectation that anybody wishing to incorporate an open source project into their own offering will ask the lead of that project for permission.

One of the dangers that commercial exploitation brings to the open source community is that the norms may be trampled upon. Is a company backed by outside investors likely to take the project owners views in mind when they have their own shareholders to concern themselves with. I’d like to be a fly on the wall in the board meeting when the VP of engineering explains to the board that a certain path cannot be followed because an open source project owner hasn’t agreed to their work being used, especially when the company would be perfectly within their legal rights to use it.

If a project lead sees their project being exploited by open source companies will it become a motivator to improve the software or will it become a disincentive?

Conclusion

Whilst there are many issues surrounding open core as a business strategy, it cannot be denied that an awful lot of high quality open source software has been written in pursuit of it.

When one surveys the open source network management landscape from before the open core invasion, it is hard to see how the user community has lost out.

I have a confession to make: I’ve developed a failed open source project! There I’ve said it, it’s now public knowledge and I can hang my head in shame… lead me to the village stocks so you can all throw rotting vegetables at me.

Happily, I don’t feel like that. Failure is, well, no big deal. Of course it does sting a little bit that I wasted an awful lot of time developing the software. What could I have done with the time had I not written the 11,184 lines of code Ohloh says I wrote? Well, I’ll never know, but…

After having a failure, any failure, it is quite healthy to take a look at it and try to figure out what mistakes were made and see if there are any lessons to be learnt.

The main mistake was to write TimeTag at all. Perhaps it would help to explain why I wrote TimeTag in the first place.

TimeTag was intended to kick start an open source network management / systems administration software ecosystem based around the PowerShell environment. (If you don’t know what PowerShell is, there is a good explanation here.)

If you want to build an ecosystem like the Linux network management / sys admin toolset, you are going to need the basic tools available to build upon. The Linux ecosystem has a few hub projects upon which most of the rest of the ecosystem build. It stands to reason that if you don’t have the hub tools, the ecosystem won’t take root.

So that’s where TimeTag came into the picture, it was my feeble attempt to build one of the hub tools for the PowerShell environment. The problem is that the Linux ecosystem didn’t develop in the way I envisaged the PowerShell environment developing.

RRDTool is the considerably more successful cousin of TimeTag. RRDTool was not written before the tools that depend upon it. Tobi Oetiker, the original author of RRDTool, also created the MRTG project. MRTG is, if not the first, then pretty close to the first, open source network monitoring application. The MRTG project originally had a very simple time series database (a mechanism for storing readings). As time went by, and MRTG was used on ever larger networks, the simple time series database didn’t scale well. RRDTool was written to provide a scalable time series database to cope the ever increasing demands placed upon MRTG.

So, rather than building one of the hub projects (a RRDTool equivalent) I should have started by building a MRTG equivalent for PowerShell instead. Then, if that had been successful, I should have written TimeTag. Instead of founding a project it would probably have been better to join the PolyMon project as a developer and then extracted the time series database into PowerShell.

As I did last year, I’ve compared the number of searches for the project name using Google Trends. As always, this post is not intended to be indicative of the usefulness of a particular tool to your requirements.

Open Source Network Management System Trends

Firstly a comparison of the major players in open source network management: Zenoss, Hyperic, Nagios, MRTG and OpenNMS. The most striking thing about the graph to me is the decline in searches for Nagios. From the middle of 2009 things have been declining quite steeply. MRTG has been declining though it just looks like a continuation of the decline evident for the last few years.

Open Source Network Management System Trend 2009

A Comparison of the Nagios Ecosystem

Whilst the above graph showed a reduction in the relative number of searches for Nagios, perhaps the Nagios ecosystem graph can explain it. Icinga, a Nagios fork, was created during 2009 and may be responsible for at least some of the decline. Icinga appears on the graph during late April and has a steady presence throughout the rest of 2009 save for a small period during the Christmas break.

A Comparison of the Nagios Ecosystem 2009

Open vs Closed Network Management Systems

Given that 2009 was a year of recession in many countries, perhaps it won’t surprise too many to see so many of both the commercial open source and proprietary tools trending downwards. I suspect that 2009 was a tough year for winkling money out of IT budgets.

Open vs Closed Network Management Systems 2009

Conclusion

All in all an interesting year. Apart from the Icinga/Nagios episode it seems odd that none of the tools has made significant progress during 2009. If open source tools were to make a move against their proprietary cousins you would assume it would be 2009 given the economic background. Budgets have been tight, so why haven’t open source tools made progress in these recessionary times?