Metric: Bug Frequency

How many bugs are there? What’s the quality like?

Quality is more involved than simply counting bugs. However, the number of bugs found over time is a reasonable indicator of quality.

“Development Bugs” are bugs that are found during development – the more of these the better as they’re found before the product gets to a customer. In contrast “Escaped Bugs” are found once the product is live in user environments which is bad. If your data can differentiate internal vs. escaped bugs you can get a sense for how effective your quality practices are at bug hunting.

We track the number of internal and escaped bugs created over time. Generally speaking we can observe a standard patter of low escaped bugs and high development bugs. If the number of escaped bugs is trending up then there may be a quality problem, even if this correlates with an increase in user numbers. If the number of escaped bugs is trending down, then quality practices are being effective.

Development bugs will fluctuate over time, but excessive numbers indicate that more quality work may be required to prevent bugs rather than catch them afterwards. We’ve observed that the number of bugs (development and escaped) reduces when development and test effort is merged and increases when there are separate development and test teams.