Code Coverage: What Is It Good For?

I had a mini-exchange with my pal Glenn Block last night on Twitter. He was recycling the somewhat old meme that code coverage is a cockamamie metric that, as he puts it, ” a false security blanket on quality”.

Measurements (other than net profit, of course) are merely indicators. They give us insight into what we’re doing, red flags that help us know when to follow up on and drill into with more qualitative and hands on tools.This is, more-or-less, the idea behind Autonomation or “automation with a human touch.”

For example: while code coverage isn’t proof that our testing efforts will yield higher maintainability, it does tell us a teams commitment to a test practice.

I tend to believe the most interesting indicators are composite metrics. What happens when we start thinking about combining code coverage with cyclomatic complexity? I’d like to read a (hopefully) inverse relationship over time, coverage rising while complexity falls, as a good indicator that we’re doing simple design.

The important thing to remember about metrics is they are only single purpose tools which, correctly applied, gift us with little tidbits of insight. It’s up to you to find meaningful and creative uses for the various metrics and their various combinations. Once you start doing that, measurements like code coverage can certainly help you achieve your immediate goals.

10 Responses to Code Coverage: What Is It Good For?

I second the comment by a few previous readers that code coverage tools can reveal some important insights, including:

* The ‘riskiest’ areas in your codebase: A combination of low coverage and high compexity definitely warrants investigation.

* Loss of code coverage over time is an indicator that code is changing but tests are not, which can be problematic.

IMHO, much of the negativity towards code coverage metrics is due to their misuse. The statistics can easily be used to instill a false sense of security in stakeholders who don’t understand that touching a line of code w/ a test doesn’t make it bug-proof. I’d prefer to see code coverage metrics as a tool used within the development team to prioritise where to invest in new and updated unit and functional tests.

Composite metrics are definitely the way to go. The more different ways you are able to measure your code quality the better. I’ve been playing with integrating NDepend and NCover into Teamcity and have come up with some nice visual dashboards:

Like anything, I think metrics are best used and studied and moderation. When coming on to a new project, you can glean a lot of insight just by getting code coverage. It helps to diagnose problems, not resolve them.

Some people I know are perfectionists when it comes to this metric. 100% code coverage in and of itself is kind of a myth – you still have integration issues as well as potential false positives with mocks and stubs. You described it really well when you said it can act like a “security blanket”

I don’t put much importance on the coverage “metric” but looking at code coverage as a whole is certainly important. Identifying areas that aren’t covered is critically important, but also the number of times each element is “touched” can be equally important especially when CC is unavoidable. The metric may say that nearly every line is touched, but that certainly lends nothing to assessing whether each combination has some form of coverage.

Bad tests can spoof any effort to mine information out of coverage so it’s certainly *not* enough to just say “The code is safe, the code is clean, so long as those bars are mostly green.” Personally I review as much of the code, and unit tests as time affords me.

Hearing about this the last couple days I realized that it’s not really the coverage number itself that I usually pay attention to but rather what it’s doing. When it’s falling I wonder what’s happening. When it’s rising I feel a bit more comfortable.

It also helps that I fully trust the testing capabilities of my coworkers.

Right, it’s a metric. Complaining about the information it gives you is like complaining about temperature not giving you the position and velocity of all molecules. That’s the point of the metric: to reduce the view on a system to a single value. It does not characterize the system, it measures something. Only by adding other metrics can you get a more real idea of the underlying system. The difficulty is to know what and how much you need to know.