Saturday, December 03, 2005

When Newton "uncovered" the natural laws of motion he was just formalising something that was known but not formally understood at that time.

The software industry is going through the same "uncovering" or natural laws and XP is our Newton.

During many years the practices advocated by XP (Integrate often, Unit test, Collective code ownership, spike solutions, Refactoring, etc.) have been successfully used with excellent results in some projects.

These practices stem from practical experience, from empirical study or "guess and test" one of the important features of the scientific method in general"[1].

Through experience the software industry has found many practices that did not work reliably (like the waterfall software development method), and has found others that work (Incremental and iterative software development method). However one basic problem prevented it from understanding the need for practices that can reliably contribute to successful software development. The economics were not in place.

As explored in this article the software industry is still very young. Many successful companies are actually run by people that had been able to strike it rich with one successful product, additionally, due to it's success (remember the IPO's of the late 90's?) the software was invaded by "want-to-get-rich-fast" MBA graduates. These people did not necessarily understand anything about software development, but were (and still are) convinced that their recently acquired business knowledge would cover for the lack of technical experience. Never mind the fact that they never created a computer program or used any other software than Excel, Word and PowerPoint. It was far to easy to start an IT business at that time, there was no motivation to increase productivity.

As luck would have it the bubble burst in 2000 and many companies actually went bankrupt. This was, of course, not a happy thing for many people, but it was a needed slump that would help renovate the industry.

After the bubble burst the MBA's were the first to leave the ship and the techies the only ones left. New products started emerging, the Internet age really started (when was the first time you heard of Google? or Blogging?).

Another positive impact in the industry was that the techies that were left after the bubble burst did not have any MBA CEO's to fight in order to implement more productive software development methods in their companies.

Around 2001 some people that had been doing things differently for a while stopped to create what is now known as The Agile Manifesto.

The Agile Manifesto described 4 values that should guide all software development to make it agile (more productive, reliable and adaptative). These people really understood what software development was all about, they had experienced it, just like Newton did when he wrote the Natural Laws.

Extreme Programming was one of the Agile Methods that brought to the forefront many of the practices that are now part of any agile project, but were at that time derided as "wrong" by the very same MBA's that eventually abandoned the industry.

These practices along with a few other are what we could call the Natural Laws of successful software development. Of course you can succeed without them, but you are most likely to succeed with them than without them.

Even before the Agile revolution had started there were already some natural laws of software development:1. Version Control2. Bug/issue tracking3. Testing

You could succeed without these but it was unlikely.

Now the field has changed, more productive methods have and will continue to change the industry. You can still succeed if you only follow the practices outlined above, but it is unlikely that you could be as successful as a project that would use:1. Prioritized requirements2. Continuous integration3. Delivering often and incrementally4. Unit testing (or Test-Driven Development if you want to go a step further)5. Pair programming or Code Reviews6. Collective Code Ownership7. Refactoring8. ...

These practices are in no particular order and it may even be that they are not enough. But these have certainly been added to the list of Natural Laws in software development.

I, for one, would not consider starting a new project without considering at least a subset of these practices.

A part of the reason must be that convincing experiments are very difficult to set up. It would be great to do a real-world software project twice with two competent teams using different methods, but the costs would be high and typically there would be confidentiality issues as well.

The empirical agile studies that I know of are typically done on teams of students programming on exercises. Let's hope that Agile-ITEA comes up with convincing results.

Thanks for the feedback Timo.Indeed I also believe that Agile methods are only one step towards perfection in software development.

I also believe that soon we will start having enough people interested in it's success (or failure) as to have more research into it.

However my goal is not to prove "scientifically" that Agile methods are better, rather my goal is to explore why they are better from an empirical perspective, i.e., I "feel" they are better. Why do I feel that?

There's an analogy with Newton's work because he did not know even a tiny fraction of what we know now about our world, but he could still formulate simple laws that survived for a long time and were _actually_ useful to many people (including me in high-school! :).

Links to this post:

About Me

Product Manager, Scrum Master, Project
Manager, Director, Agile Coach are only some of the roles that I've
taken in software development organizations. Having worked in the
software industry since 1997, and Agile practitioner since 2004. I've
worked in small, medium and large software organizations as an Agile
Coach or leader in agile adoption at those organizations.

I was one of the leaders and catalysts of Agile methods and Agile culture adoption at Avira, Nokia and F-Secure.