Coping with Bugs on an Agile/Scrum Project

An often asked question is how does Scrum recommend a team to handle bugs? Should they be placed on the product backlog? Or on a separate bug list? If they’re on the backlog, does the Product Owner get to set their priority or are they automatically the most important items? Should there be a separate bug fixing sprint?

On Pascal Maugeri’s team, even after improving their definition of done and doing “proper testing/unit-testing”, they’re still finding bugs that escape the sprint. He asks how to resolve this problem.

George Dinwiddie, Agile Coach, suggests raising the issue with the team during the retrospective – he has worked with teams that had vanishingly small bug rates. Mark Levison (this reporter) suggested “I would start to ask the question why weren't they found and fixed within the sprint that they occurred? My focus is on reducing the time it takes to discover (and then fix) problems. After all, if we discover a bug in the story during the sprint it is written then the PO shouldn’t accept it as being done. In addition early discovery will make it much easier to fix as the code is still fresh in the mind of the development team.”

Jim Schiel, CST with Artisan Consulting, just puts bugs in the product backlog to be prioritized by the Product Owner - “unless it’s a dead simple fix, in which case you determine the solution during Sprint Planning and execute the solution during the Sprint.”

Bruce Kantelis says it’s all about developing a culture: “We categorize defects. Priority 1, blocking the user from working get immediate attention, the interrupt work for a fix and patch, all others become stories that are placed at the top of the list for the next sprint. Over time, teams realize that quality measures and behaviors really impact their workday and rally to minimize the interruption.”

Mike Cohn reminds us that bugs found within the sprint are best handled by yelling across the team room and describing the bug. Failing that, a card describing the bug can be added to the board. However for bugs that escape the sprint – he prefers to add them to product backlog, leaving it up to the product owner to prioritize them. Many existing teams still have a bug database that they need to continue using. In that case, he counsels maintaining a separate bug backlog, where product owner simply says what his priorities are from each queue: i.e. the first two items from the product backlog, then the following bugs and finally the next two items from the backlog.

Kevlin Henney argues that this approach is akin to treating a bug as a feature with a negative value:

If defects are viewed as features with negative value, they become managed as if they were features. Development groups store up prioritized repositories of bugs, treat bugs as user stories, outsource the fixing of bugs, and so on. While each of those can be a useful technique or perspective in dealing with a project in transition or crisis, it is not a long-term view that should be encouraged. After all, as the "Manifesto for Agile Software Development" says, "Working software is the primary measure of progress." It is a little disingenuous to pass off a feature with known defects as complete and working — "Yes, it's done... but there are a few bugs."

Ron Jeffries turns the whole problem on its ear saying that fixing a defect in a feature after it's finished is always more expensive than doing it right the first time.

So when we write the software incorrectly and then fix it, the customer pays more: she pays whatever she paid before, plus the bug fixing penalty.

She should be really ticked off about that. I like to encourage the customer to prioritize all the defects, to ensure that she'll feel the pain of the team's inadequate software process. I feel sure that she'll express that pain and that the team has a better chance of getting the idea that doing things well is better.

Do you avoid bugs altogether? Put them on the Product Backlog? Do you see the problems Kevlin outlines?

Of course, we try to minimize bugs at the origin, so at development time. Nevertheless, since no person and no methodology is perfect, and especially since we're doing distributed development in India, which doesn't permit to follow up everything adequately, some bugs might get through to the 'internal release', which will be tested by our testing team in Brasil.We make use of a bug tracker system to post the issues and give them a degree of severity.Now, the development team just takes 3 days every sprint (4weeks) to solve issues. So they have a pick from the most severe ones, complemented with some minor, easy fixable stuff. This every sprint again. This seems to work for us. Although, sometimes, very major issues, which are mostly design related, may need to be placed on the Product Backlog, because it actually is a requirement change or rewrite, but we only had to do this once until now.

There's a guiding principle that everyone should apply - "Bug driven development is evil". But it's common. And bugs on the backlog are a varient of bug driven development.

Why is it evil? Because bugs are impossible to estimate. OK, maybe impossible is too strong, but I'd argue that "several orders of maginitude harder to estimate than features" is accurate. If you plan your project using a list of bugs then your estimated timescales are going to be utterly useless.

What's the antidote? What's the opposite of "bug driven development"? It's much easier to estimate how long it will take to test something than it is to estimate how long it will take to fix bugs in it. So if you've got high priority bugs that have escaped the sprint then put tests for them on the backlog. It's not the conventional meaning of "test driven development", but it works for me.

#, First, to reiterate, do everything you can to never let the code leave the developers desk with bugs. Duh.

Second, if you find it during the iteration, fix it. Verbal communication to coordinate this is best.

Finally, if it's found post iteration - fix it somewhere between "now" and "first thing next iteration" (at the latest), favoring "now" whenever possible, depending on the criticality of the bug.

Consider this: if you could have predicted the bug when you first slated the story for development, would you ever have considered prioritizing that bug for "sometime later"? Of course not, you'd have demanded the bug never make it in. Why allow a different attitude when the bug comes in later?

Bottom line, keeping growing repositories of bugs (whether in bug trackers or backlogs) is a bane to productivity. Make it a top priority to avoid.

My take is this: if you have a "bug", meaning the software aint doing what was expected, then it either gets fixed ASAP (see my other response), or it's not a "bug". If it can "prioritized for later", then it becomes a real-deal story.

I think this approach is on the money and in the long run you will have as George said "vanishingly few bugs". Most teams I see shy away from this approach saying but we will never get any work on features done. Sad. Keep at this Anton you're going in the right direction.

I agree with the ones of you guys saying, that bugs should never escape the developers during a sprint, as long as we are talking green field projects, or at least projects that has been running scrum from the beginning.

In "brown field" projects, and projects having recently made the switch from something else to scrum, it's a very different ball game.

Typically projects like that are blessed with a large number of known bugs, and perhaps even more unknown ones. Bug tracking systems are often in place, and the customer is acustomed to discovering new bugs within every release. Unit tests are absent or at least way too few, and testers are often a scarce resource, so it's only on rare occasions, that the entire system is tested properly.

On projects like these, it is often not feasible to have a strategy to fix a bug as soon as it is found, because of the hard-to-estimate nature of bugs - the team will not get any features "done" in the sprint.

If the customer is paying per implemented feature, and not for bug fixes, it actually works quite well to treat bugs as "features with negative value", as Kevlin puts it. That way the customer can prioritize wether to have the team focusing on stabilizing the product or adding new functionality.

I would strongly recommend teams on projects like this apply the techniques explained by Michael Feathers in the book "Working Effectively with Legacy Code", so the code coverage of the system increases sprint by sprint.

By the way we have a joke with my colleague "If you want to write software do it at home not in the office!". The reason is that nobody wants to invest time in what ever activity which doesn't return value now ! ;-)

You hit the nail on the head. This is the first question you must ask... is it a bug, or is it a change? I've seen a lot of bugs that have come up that were "We asked you for x, never thinking about y, so could you please change the system so that y is covered?" It's a business scenario, so there's little reason to expect a dev or tester to anticipate/test it. I know where we've struggled is in injecting items into the product backlog, we tend to classify it as a bug and so we end up with a lot of bugs, but not a lot of change in the back log. That's exactly what I think we're supposed to be avoiding in Scrum or any other agile methodology. As we see change, we should be taking it on as a feature and prioritizing it accordingly. Only when it's something broken should it be called a bug.

At my last company, we took the following approach (the app was a single-instance SaaS serving about 2.2 million K-8 students and their teachers):

* Each release (approx 3-4 2-week sprints) a team member was assigned to the 3rd-level support role. The rest of the team focused solely on the current release.

* The person assigned to 3rd level support was permitted to work on applied R&D projects of his/her choosing that we agreed were worthwhile to the company. You can think of these as "spikes" to increase stored value in using new technology we were considering to solve problems further out on the roadmap. Sometimes it was simply paying down any "technology debt" that had accrued.

* These projects had to be shelved immediately if any defects appeared in production and were affecting customers. The person was permitted to spend up to one day without prior approval from the product owner. During that time if it couldn't be solved it had to be characterized via a clear test case and an estimate on how it impacted customers. The product manager could decide whether it became a story or whether the person doing 3rd level support could continue for a bit longer.

* Fixing the bug included identifying whether the defect was based on fragile architecture, insufficient testing, or imperfect understanding of what the customer's expectations would be. (When you have a large customer base, you find folks who use the product in a way that doesn't match the original scenarios.)

One of the benefits of this approach is that everyone has a positive incentive for minimizing defects - more time to pursue ideas that increase the team's velocity significantly when it's "your turn" to take the 3rd level support role.

One shouldn't classify things into bugs vs features, in first place. It leads to wasted time spent arguing whether something is a bug or not, make people feel and treat them differently, and is awkward to manage, as, after all, it takes exactly the same work to do either.

What exists is an unsatisfied requirement. If the user says the requirement is part of something that was already done, then you pick the whole story, bring it back to the product backlog, do whatever further detail the team thinks necessary -- if they think it necessary, and prioritize it right along everything else.

That gets rid of the blame game, focus on the fact that the requisite is not "done", and makes it possible to prioritize workload appropriately.

Now, if that is not viable because the contract says bug correction is free, everything else is not, then you have a bad contract. Instead of trying to get someone else to pay up for badly written software -- since you'll not get that anyway, the price tag will just be added to other expenses one way or another -- you should focus on getting good developers, good tools and good process.