Working on the "technical matters" works best when it is driven by stories. The code base is probably in need of work everywhere, but the payoff will be received only where the code is going to be worked on for user-facing reasons. If no stories are going to pass through some crufty area, working on it is largely wasted.

Therefore, I prefer the approach of taking stories as usual (but probably fewer of them), and following the "boy scout rule" of leaving the campground better than you found it. In other words, wherever the stories lead us, let's write more tests, let's refactor more aggressively.

This approach has at least these advantages:

maintains "best sensible" flow of stories;

provides help from all team talent;

provides for whole team to learn how to keep code clean;

focuses improvement exactly where it is needed;

does not waste improvement that "may" be needed;

I've seen code quality have a very big effect on long-term productivity, so I am a believer that technical debt should be taken care of. I think the post above makes sense, but I'm not so sure about the last two points. I'm interested in finding out real experiences of benefits from cleaning technical debt, even if it was not related to user stories.

What positive benefits have you seen from cleaning up your code base and ridding yourself of technical debt? What methods did you use to get the work done?

Why would code even exist, if it doesn't affect a user story? (administrators of a system are still users - hence logging and 'under the covers' stuff still applies)
–
Steve EversDec 10 '10 at 1:01

2

@Sn0rfus That's a good point. I have however worked with teams that refused to reconsider whether anything that was considered "working" was done correctly. These would never get cleaned up because the features were considered "done". They would often have huge indirect effects on future development because they were done poorly, but developers and our manager alike would simply turn a blind eye.
–
NickCDec 10 '10 at 4:48

6 Answers
6

About 10 or 12 years ago I inherited an application from a team of developers that ended up leaving the company (too long to get into here...). The system was a large home-grown middleware report generation system. It ran every week night and generated about 2 dozen Excel reports for senior executives of a Fortune 500 company. When I inherited it, it took about 5-6 hours to run and during any given week would fail at least 2 nights.

I was not a happy camper to be given this mess.

Initially my plan was just to stop the bleeding and fix the main cause of the failures. After I became more comfortable with the code base, I started looking for places where I could refactor and add stability and performance. Over the course of 2 years or so, I made many, many changes to the system. We retired that system a couple of years ago and at that point the whole process took 45 minutes to run and hadn't generated any issues in years.

A lot of work went into paying down the technical debt but it was well, well worth it. It was nice not getting any phone calls in the middle of the night that the system failed. It was nice coming into the office in the monring and seeing nothing but good news in the logs.

(Aside... After a couple of years I ran into one of the main developers of this system. He asked me how it was doing and I told him how bad the system was. He actually apologized and told me he knew it would be a handful to support after he left and wished he had done a better job on it).

It's been my experience that the benefits of code cleanup are most noticeable when I have to maintain code where the cleanup hasn't been done. Where the cleanup's been done, my changes consist of reading through the code, spotting the one or two places that need to be changed, and going from there. If the cleanup hasn't been done, add an initial step of reading the code a couple of times and trying to figure out what the author (sometimes me) was thinking when he wrote it.

always

This is not necessarily true. The last two bullet points in the OP's comment mean that you should not work on refactoring willy-nilly. If you find that a rarely used piece of code is very poorly written and you decide to eliminate that technical debt, that means that you cannot add new functionality or remove technical debt somewhere else, say somewhere that IS used a lot. The reality is that we have limited time and absolutely have to prioritize where and when we decide to remove technical debt.
–
NemiDec 15 '10 at 18:08

@Nemi: all technical debt is not created equal; please use good judgement.
–
Steven A. LoweDec 15 '10 at 19:41

1

I was just commenting, you know, because of the big bold ALWAYS in your post. I guess maybe I misunderstood your answer.
–
NemiDec 15 '10 at 22:55

One experience I had was when I was managing a Site Performance team at my previous employer. Every night, for a period of an hour to two hours, the website my team was monitoring would dip below acceptable performance thresholds due to a bot scraping information from the site rapidly. The measures the team took to address this consisted of logging into a manual admin system and blocking the IP addresses which were causing the troubles. Needless to say, this cost one member of the team hours of sleep nearly every night. I noticed what was occurring and took the on-call BlackBerry myself for several days to see how bad it was and give my team some rest.

After a few days, I simply went to the business owner of the team and let them know that if we did not implement an automated blocking system such that bots would have a much more difficult time affecting the performance of the site, we would likely lose some if not all of the team members due to fatigue and burnout. They agreed, and we implemented a system which allowed us to get sleep at night. The business owner understood that the cost of a few days or a week of development was minimal compared to the cost of hiring/training new engineers.

+1 for discussing the problem with the PO/BO. That's how it should work (ideally :-)).
–
sleskeNov 14 '11 at 17:13

And BTW, I would not even call that an example of technical debt. This is clearly a missing feature, which your team had to compensate for by manual work. My definition would be: If it affects the end user (directly or indirectly), it's not technical debt, but simply a bug/missing feature
–
sleskeNov 14 '11 at 17:16

Regarding the last two points: I understand where it's coming from, as explained in his original post :

Or, is it possible to reassign some developers to get these
technical matters done, while the rest of the team continues on
the user-oriented stuff? This may affect team velocity, but so what?

"So what" equals: the product owner
and other business-side people become
unhappy. And when Momma's unhappy,
everyone's unhappy.

Yet, the line between what has to be done and what may be done is pretty vague. User-faced is very broad, and includes performance and occurence of errors. But in quite some cases, the underlying problem of poor performance and high occurence of errors lies deeper in the code. To say it in his words: a story might not go through a crufty area, but that crufty area can hide some nasty things that attacking the story on the cleaned-up path next to it.

Things that do not influence the overall performance are less interesting to clean up, but one should evaluate very carefully the influence of those points. More often than not they have an indirect influence that can be pretty substantial.

The biggest benefit an organization will receive as a result of paying down technical debt is avoiding the compound interest. There is an example in the blog entry below that shows how the principal amount owed on a technical debt went from $160k to $430k in just five years. It would take a full time programmer solely dedicated to servicing that amount of debt. That will help put it in perspective for the decision makers!