About

@dailyshoot

Archive for the ‘General’ Category

“Just like Arthur Dent, who after inserting a Babel fish in his ear could understand Vogon poetry, a computer program that uses Tika can understand Microsoft Word documents.” This is how Tika in Action, our book on Apache Tika, introduces it’s subject. Download the freely available first chapter to read the the full introduction.

Chris Mattmann and I started writing the Tika in Actionbook for Manning at the beginning of this year, and we’re now well past the half-way post. If we keep up this pace, the book should be out in print by next Summer! And thanks to the Manning Early Access Program (MEAP), you can already pre-order and access an early access edition of the book at the Tika in Action MEAP page.

If you’re interested, use the “tika50” code to get a 50% early access discount when purchasing the MEAP book. You’ll still receive updates on all new chapters and of course the full book when it’s finished. Note that this discount code is valid only until December 17th, 2010.

We’re also very interested in all comments and other feedback you may have about the book. Use the online forum or contact us directly, and we’ll do our best to make the book more useful to you!

Update: The book is out in print now! Use the “tika37com” code for a 37% discount on the final book.

Open sourcing a closed codebase can be difficult. The typical approach is to decide that you’ll go open source, make big news about it and then try to figure out how to proceed. It’s no wonder many open source transitions end up being more painful than expected and fail to generate as much community interest and involvement as hoped. How can you do better?

0. Start small

Even though your marketing people will be eager to use a good story, you should to avoid the temptation to make a big deal about your shiny new open source project. Instead, start with small, reversible steps that allow you to get comfortable with the new way of developing software before making public commitments. In other words, learn to walk before you try to run. The next sections outline how to do this.

1. Clean up the codebase

Do you really know what’s inside your existing codebase? Do you have rights to use and redistribute all the included intellectual property? Are there trade secrets or other bits in the codebase that you’d rather not show everyone? Do you wish to keep parts of the codebase closed so you can keep selling them as an add-on components on top of the open source offering?

Answering these questions should be your first task. You’ll need to spend some time auditing and possibly refactoring your code to prepare it for the public eye. Depending on the codebase this could be anything from a trivial exercise to a significant project. The nice thing is that the increased understanding and potential modularity you gain from this work will be quite valuable even if you never take the next step.

2. Open up your tools

Now that your codebase is clean and ready for the public view, you can (and should!) start using public tools to develop the code. You can either make your existing version control, issue tracking and other tools public, or migrate to a new set of public tools. There are plenty of excellent free hosting services for open source projects, so you have a good opportunity to both lower your maintenance costs and improve your productivity through better tooling!

There’s no need yet to worry about external users or contributors. In fact the fewer people you attract at this stage, the better! The main purpose of this step is to make your developers comfortable with the idea that anyone could come and see all their code and all the mistakes they are making. This is a big cultural change for many developers, and you’ll want to start small to give them time to adapt in peace.

3. Engage the community

If you’ve followed the steps so far, you’ve actually already open sourced your codebase. Are you and your developers comfortable with the situation? It’s still possible to switch back to closed source with minimal disruption and no lost reputation if you’re having second thoughts. But if you are willing to move forward, now is the time to start enjoying the benefits of open development!

Call in your marketing people to do their magic. Tell the world about the code you’re sharing, and invite everyone to participate! If you’re product is in any way useful to someone, you’ll start seeing people come in, ask questions, submit bug reports and perhaps even contribute fixes. At this point it is useful to have a few people ready to help such new users and contributors, but it’s surprising how quickly the community can become self-sufficient. More on that in a later post…

There are many different ways and reasons for companies to develop their software as open source. Here’s some brief commentary on the main approaches you’ll encounter in practice.

0. Closed source

Well, closed source is obviously not open, but I should mention it as not all software can or should be open. The main benefit of closed source software is that you can sell it. If you are working for profit, then you should only consider open sourcing your software if the benefits of doing so outweigh the lost license revenue.

1. Open releases

Also known as code drops. You develop the software internally, but you make your release available as open source to everyone who’s interested. Allows you to play the “open source” card in marketing, and makes for a great loss leader for a “pro” or “enterprise” version with a higher price tag. And no changes are needed from more traditional closed source development processes. Unfortunately your users don’t have much of an incentive to get involved in the development unless they decide to fork your codebase, which usually isn’t what you’d want.

2. Open development

Making it easy for your users to get truly involved in your project requires changes in the way you approach development. You’ll need to open up your source repositories, issue trackers and other tools, and make it easy for people to interact directly with your developers instead of going through levels support personnel. Do that, and you’ll start receiving all sorts of contributions like bug reports, patches, new ideas, documentation, support, advocacy and sales leads for free. You can even allow trusted contributors to commit their changes directly to your codebase without losing control of the project.

3. Open community

Control, or the illusion of it, is a double-edged sword. If you’re the “owner” the project, why should others invest heavily in developing or supporting “your” code? To avoid this inherent limitation and to unlock the full potential of the open source community, you’ll need to let go of the idea of the project being yours. Instead you’re just as much a user and a contributor to the project as everyone else, with no special privileges. The more you contribute, the more you get to influence the direction of the project. This is the secret sauce of most truly successful and sustainable open source projects, and it’s also a key ingredient of the Apache Way.

So what’s the right way?

There’s no single best way to do open (or closed) source, and the right model for your project depends on many factors like your business strategy and environment. The right model can even vary between different codebases within the same company. For example in the “open core” model you increase the level of innovation in and adoption of your core technologies by open sourcing them (or using existing open source components), but you make money and maintain your competitive edge through closed source add-ons or full layers on top of the open core. This is the model we’ve been using quite successfully at Day (now a part of Adobe).

If you’ve decided to go open source and you don’t have a strong need to maintain absolute control over your codebase (like I suppose Oracle now has over the OpenJDK!), I would recommend going all the way to the open community model. It can be a tough cultural change and often requires changes in your existing development processes and practices, but the payback can be huge. In military terms the community can act as a force multiplier not just for your developers, but also for the QA and support personnel and often even your sales and marketing teams!

If you’re interested in pursuing the open community model as described above, the Apache Incubator is a great place to start!

I find it interesting that many of the above stories give the impression of Chongqing as a megacity of 30+ million people, when in fact (or at least according to Wikipedia) the urban population is “just” 5+ million people and a majority of the rest are farmers living in the surrounding areas that are administratively part of the city.

Such comparisons are of course highly debatable, as seen in the comments that Michi’s post sparked, but for the fun of it I wanted to see what the equivalent code would look like in Clojure, my favourite post-Java language.

I guess the consensus among post-Java languages is that features like JavaBean-style structures and functional collection algorithms should either be a built-in part of the language or at least trivially implementable in supporting libraries.

The news is just in about Adobe being set to acquire Day Software (see also the FAQ). Assuming the deal goes through, it looks like I’ll be working for Adobe by the end of this year. I’m an open source developer, so I’m looking forward to finding out how committed Adobe is in supporting the open development model we’re using for many parts of Day products.

The first comments from Erik Larson, a senior director of product management and strategy at Adobe, seem promising and he also asked what the deal should mean for open source. This is my response from the perspective of the open source projects I’m involved in.

First and foremost I’m looking forward to continuing the open and standards-based development of our key technologies like Apache Jackrabbit and Apache Sling. There’s no way we’d be able to maintain the current level of innovation and productivity in these key parts of our product infrastructure without our symbiotic relationship with the open source community.

Second, I’m hoping that our experience and involvement with open source projects will help Adobe better interact with the various open source efforts that leverage Adobe standards and technologies like XMP, PDF and Flash. The Apache Software Foundation is a home to a growing collection of digital media projects like PDFBox, FOP, Tika, Batik and Sanselan, all of which are in one way or another related to Adobe’s business. For example as a committer and release manager of the Apache PDFBox project I’d much appreciate better access to Adobe’s deep technical PDF know-how. Similarly, in Apache Tika we’re considering using XMP as our metadata standard, and better access to and co-operation with the people behind Adobe’s XMP toolkit SDK (see more below) would be highly valuable.

It would be great to see Adobe becoming more proactive in reaching out and supporting such grass-roots efforts that leverage their technologies. I’ve dealt with Adobe lawyers on such cases before with good results but it did take some time before I found the correct people to contact. Another area of improvement would be to make freely redistributable Adobe IP more easily accessible for external developers by pushing them out to central repositories like Maven Central, RubyGems or CPAN, for example like I did when making PDF core font information available on Maven Central.

Finally, it would be great to see Adobe going further in embracing an open development model for some of their codebases like the XMP toolkit SDK that they already release under open source licenses. I’d love to champion or mentor the effort, should Adobe be willing to bring the XMP toolkit to the Apache Incubator!

Ceki Gülcü recently wrote an interesting post on the Apache community model and its vulnerability in cases where consensus can not be reached with reasonable effort. Also the discussion in the comments is interesting.

Ceki’s done some amazing work especially on Java logging libraries, and his design vision shines through the code he’s written. He’s clearly at the high edge of the talent curve even among a community of highly qualified open source developers, which is why I’m not surprised that he dislikes the conservative nature of the consensus-based development model used at Apache. And the log4j history certainly is a sorry example of conservative forces more or less killing active development. In hindsight Ceki’s decision to start the slf4j and logback projects may have been the best way out of the deadlock.

Software development is a complex task where best results are achieved when a clear sense of architecture and design is combined with hard work and attention to details. A consensus-based development model is great for the latter parts, but can easily suffer from the design-by-committee syndrome when dealing with architectural changes or other design issues. From this perspective it’s no surprise that the Apache Software Foundation is considered a great place for maintaining stable projects. Even the Apache Incubator is geared towards established codebases.

Even fairly simple refactorings like the one I’m currently proposing for Apache Jackrabbit can require quite a bit of time-consuming consensus-building, which can easily frustrate people who are proposing such changes. In Jackrabbit I’m surrounded by highly talented people so I treat the consensus-building time as a chance to learn more and to challenge my own assumptions, but I can easily envision cases where this would just seem like extra effort and delay.

More extensive design work is almost always best performed mainly by a single person based on reviews and comments by other community members. Most successful open and closed source projects can trace their core architectures back to the work of a single person or a small tightly-knit team of like-minded developers. This is why many projects recognize such a “benevolent dictator” as the person with the final word on matters of project architecture.

The Apache practices for resolving vetos and other conflicts work well when dealing with localized changes where it’s possible to objectively review two or more competing solutions to a problem, but in my experience they don’t scale that well to larger design issues. The best documented practice for such cases that I’ve seen is the “Rules for revolutionaries” post, but it doesn’t cover the case where there are multiple competing visions for the future. Any ideas on how such situations should best be handled in Apache communities?