Development Psychology, Technical debt and The next feature syndrome

I would like to describe
here a harmful scenario I have seen in many different occasions. Typically, a
software is built through iterations and each iteration is an occasion to add
new feature(s). At the beginning of a project, each new release represents a
giant step in terms of features. This is because the product is young and is
missing important features. At that time, the team is working hard to make
sure that users don’t miss important features. Each iteration is pressured by
the feature(s) it will provide. This pressure fosters the team to increase itstechnical debt
as described by Martin Fowler: You have a piece of functionality that you
need to add to your system. You see two ways to do it, one is quick to do but
is messy – you are sure that it will make further changes harder in the future.
The other results in a cleaner design, but will take longer to put in place.

Typically the technical debt
includes:

Absence of tests and/or poor
test coverage ratio

Poor design (mainly dependency
cycles between components or even worst, no clear componentization at all)

After a few years
however, if the product didn’t die it means it entered a cruise phase: It
satisfies its users and it should be comparable to competitors in terms of
features. At that point, managers and developers have divergent opinions about
what should be done:

Developers get sick of
working on a code base with a lot of technical debt. The code is hard to
maintain, bugs are hard to fix and each new feature is a nightmare to implement.

Managers want to make
more money with the software and estimate that the energy must be focused on
adding new features to sale a more competitive and richer product.

Personally I have never
seen any manager accepting stopping adding new features for a long time to
repay the debt. This is a good thing because to survive software needs new features as
human needs oxygen. However, I have seen many times the
following dangerous scenario:

Managers don’t want to
hear anything about the technical debt and they keep pressuring the team with
new features. This reflects a mid-term view and sooner or later the code base
will collapse. Concretely collapsing mean that at a point it will take days to fix bugs and months
to add new features.

Developers are sick to
work on a dirty old code. They suggest managers that it could be cheaper to re-code
everything from scratch while maintaining the old code base. Managers are first
reluctant but end up by agreeing the necessity for this v2.

Quickly the v2 takes 80%
of the development effort because it is more fun and considered more important, but during that time the
v1 is making 100% of the income. This situation is extremely dangerous because
users might be bored to use a product that is not evolving (the v1) and might prefer
switching to competitors. In addition the team has the illusion that the
product is evolving since indeed, v2 is evolving.

I am sure that if you,
reader, are not currently living this scenario, at least you faced it during your
precedent projects. The only way to avoid entering this nightmare is to keep
the debt small while adding new features, as soon as possible in the life of
the project. Hence the importance of short iterationswhere
it is easy to fix and achieve concrete goals in terms of feature and technical
debt repay. Hence the importance of Evolutionary Designto constantly restructure and refactor the code
base to adapt it to new constraints. Hence the need to say at a point: From Now all code added or refactored will have high tests coverage ratio,
and will have a high quality.
Let’s stop increasing the debt.

IMHO you should always avoid
the temptation to start a v2 from scratch. Take the time to re-code or
re-factor the most catastrophic components, maybe migrate step by step from
your old C++/COM/VB to more recent technologies like .NET, but avoid the temptation
of starting a v2 from scratch. 100% of the improvement should be done on the code base
whose currently in production. See also this Jeff Atwood post on repaying the technical debt and the fear of breaking code that works.

Great article. Refactoring is the way to pay down your technical debt but wouldn’t it be great if you could avoid technical debt altogether? Your readers might be interested in this IEEE article on why technical debt happens in the first place. I blogged about that at http://www.dynamicalsoftware.com/cgi-bin/ViewBlogEntry.pl?id=15

http://www.NDepend.com Patrick Smacchia

Steven, I think your statement is invalid, don’t recode everything just because original coders are not here anymore.

Turn over of developers should be treated as a rule in software development, not an exception.

One of the agile tenet is to be able to continue the work of someone else, to be able to recover its mental state quickly from its code, that’s why coding standard exist.

http://martinsantics.blogspot.com Martin Evans

Great post. Been there, seen it, completely agree. As a developer, the most important thing is to take a step back occasionally and look at the wider picture. When you see the “debt” begin to increase, get on the case immediately. A debt free product is a journey not a destination.

http://www.cuttingedge.it/blogs/steven/ Steven

Your statement sounds a lot like what the authors of “The big ball of mud” (http://www.laputan.org/mud/) say. However, they note a great point for starting all over: “One reason to start over might be that the previous system was written by people who are long gone. Doing a rewrite provides new personnel with a way to reestablish contact between the architecture and the implementation. Sometimes the only way to understand a system it is to write it yourself.”

However, I prefer starting to write requirements and unit tests before deciding to completely start from scratch.