Oracle Blog

databases and more

The pursuit of openness

When I joined MySQL in 2006, after several profitable years as a consultant, I had a dream. I wanted to improve the product that had contributed to my professional success.

The first thing that I learned when I started the uphill task is that it was far more difficult than expected. MySQL called itself open source, but the development practices were for all practical purposes closed source. At the same time, I found that MySQL, below the surface, is an organization with complex and well oiled engineering practices.

Indeed, opening up the cathedral, as Lenz put it, was a hard nut to crack. We had a closed source revision control system, and our developers loved it so much, that any proposal to change it was met with strong opposition. We discussed technical matters behind the firewall. Our code review process, although technically demanding and very thorough, was kept in our internal mailing lists.

Contributions were scarce, with no good signs for the future. The binary split between community and enterprise servers made bug fixes contributions very unlikely. The impossible roadmap, with contributions going theoretically to version 6.1, in the distant features, practically discouraged any new proposal of code contribution, except for a few brave souls. On top of it, the Contributor License Agreement (CLA) had only recently been set up, yet formed a formidable obstacle for many people.

Faced with such challenges, several people left. Either to new jobs inside the company or to new shores. The challenge is that you can't fix such a problem in one sweep. Changing practices in a company, especially from a position in which you can't take decisions, is really hard.

But not everyone left. The ones who stayed, in spite of the adverse circumstances, share a strong belief in the values of open source, and were determined to succeed, no matter what it takes.

With a little help of my team mates, Colin, Lenz, and Dups, I continued to support Kaj's internal action to spearhead the changes. While my younger colleagues were busy managing the community, Kaj and I were gently wrestling the rest of the company, to steer its direction towards a more open course.

The results arrived. Slowly, but surely.

In 2007, many developers opened the IRC gates, and started using freenode.net;

June 2008 - The open source Bazaar replaced the previous lock-in proprietary software Bitkeeper, which didn't allowed handling of the source from the outside;

In October 2008, we got rid of the CLA, in favor of the more open Sun Contributor Agreement.

The major obstacles were still the binary split and the long release cycle. There were much discussion internally, pleading and cajoling, to convince people that we were on the wrong path, and that our business model required a more open approach. Our team was under pressure. We were between the hammer of the community, shouting at us for not being open, and the anvil of the sales and marketing teams, unwilling to accept the assumption that a happy community generates future customers.

We kept going, and we kept pushing, until we got the announcement at the MySQL Conference 2009, stating the end of the binary split. Now the community server binaries will be published as often as the enterprise ones. The rationale of this request is that the strength of MySQl is in its wide community. We claim that the MySQL server is tested by millions of users, and yet we were giving the GA (mature) binaries only to a handful of customers.

This looked like a privilege, but it was in fact, from an engineering standpoint, a disadvantage. As a customer, I would gladly adopt a software that has been installed by a few million people, rather than being the privileged first one to try it in production.

With this announcement, we are back to the position that has made MySQL so popular and trusted. A database with a large user community.

The last barrier to fall will be the roadmap. So far, we have been tied to an impossible roadmap, with a wish list of features that had little chance of being released. The experience of MySQL 5.1, which took 37 months to release, has taught us a hard lesson. The next version, 6.0, has been in alpha for 25 months, with no near end in sight. Now we took the hard decision to replace the previous roadmap, where we piled features over features in an impossible attempt of pleasing future customers, with a milestone model, where we release features as they are mature, without waiting for all the features in the list to be ready. It means that we renounce the goal to release Falcon and Maria and the Online Backup as one single big bundle released at the same time in 6.0, and we put them aside. The MySQL engineers will schedule features for release as soon as they reach beta status, without waiting for the trailing ones.

Furthermore, we have now a process in place to accept contributions and to review them, with active participation from core developers. Yet, we have still work to do, to smooth the rough edges both internally and externally.

To our developers, who have not succumbed to the pressure, and have delivered top class software, while accepting our team's (not always) gentle suggestions to more openness, a hearty thank you!

To the many who have believed in MySQL and in our single and team efforts, thank you! To the many, who have suggested improvements, and contributed actively to our changes, thank you! To the ones who have not joined the flame war when unjustified rumors spread across the net, thank you!

To the ones who spread FUD, and twist every tiny piece of news in the shape of a conspiracy, I say wake up. Have you considered the possibility that we actually strive for the best for our community?

To the naysayers, who shout at us and abuse our goodwill from their self appointed righteousness, which often hides a secret business agenda, I say that we have always be working with the community in mind, and we will continue to improve. The path is still uphill, but the community team, the guardians of MySQL ubiquity, is happy with the progress so far, and will continue on this course!

The change of community binary frequency didn't read to me as "an end to community/enterprise binary split".
Will they use the same codebase and just have different version-tagging during the build? Right now, it's not exactly the same code for at least some of the time, and that's big part the problem - enterprise clients get to test code before the community users get it.

The announcement came too late to affect the latest release, which was already in progress. Thus, for this month only, we will still release from two code bases. MySQL 5.0.80 (Enterprise) and 5.0.81 (Community) will be however released simultaneously.
Starting with MySQL 5.0.82, the code will be merged.
Whenever a MRU is released, the same binaries will also go to the community download pages.

Jan, please have a little patience for the people who've spent years seeing MySQL spurning its community by throwing good releases after bad, and who don't feel sure they understand the change fully. Note that I've already publicly praised the 5.4 release. But I'm sorry, it's going to take actions, not words, to regain trust. MySQL needs to put its source where its mouth is. I look forward to it, but it ain't there yet.

Giuseppe, as one of the millions of anonymous mysql users, I'd like to thank you for both, this post that showed me the bigger picture of what happened and happens as well as for what you and your team mates do to regain and foster trust of the community towards mysql. At least for me it worked this way.

Arjen,
MySQL 5.1 doesn't have the problem of different code base.
We still produce different binaries with different feature sets for the convenience of our customers, but the community edition has all the features and the bug fixes of the enterprise set.

Giuseppe, you've already been thanked for explaining things in this post. I want to thank you (and Kaj!) for the work behind the scenes that put you in a position where you could write this - I know how hard it must have been.

As an Enterprise customer, I've been saying for years that what's now happening must happen, and I'm very, very glad things have turned.

I'm going to continue criticism where I see problems, but don't think your work isn't appreciated.

Thank you for open this, this is awesome you made these steps. I hope you will not be fired soon because of this post.

May I sound some more concerns, because despite of you did it still looks like MySQL is going into closed development model even more ?

Examples what I have in mind - it is 5.4 release, which seems was made and prepared in "deeply secret mode". The same for PERFORMANCE_SCHEMA which was announced few week before conference - sorry, I do not know if it published already, but it was "in closed source mode" at that time. And PERFORMANCE_SCHEMA was developed in "secret lab" without asking community for any input what we would like to see.