We are starting a push for code coverage here at my work, and it has got me to thinking.... How much code coverage is enough?

When do you get to the point of diminishing returns on code coverage? What is the sweet spot between good coverage and not enough? Does it vary by the type of project your are making (ie WPF, WCF, Mobile, ASP.NET) (These are C# classes we are writing.)

11 Answers
11

We aim for at least 70%. On things that are more easily testable (functional data structures, for example), we aim for 90% and most individuals aim for as near to 100% as possible. On WPF-related things and other frameworks that are very difficult to test, we get much lower coverage (barely 70%).

Is WPF inherently hard to test, or have you not yet spent the effort working out the best strategy to get better coverage?
–
JBRWilkinsonOct 19 '10 at 21:30

A lot of it stems from the fact that WPF input is hard to fake. Our testing is as unit-y or API-y as we can get it, and the inability to easily fake the layer that sits "on top" of WPF (input, at least) makes it difficult to test. It isn't a huge problem, as the non-GUI parts of the API are easy to test, but it's just that last stretch that goes from our model (or view model) to WPF that is challenging.
–
Noah RichardsOct 20 '10 at 2:03

1

Yeah, WPF is a bitch to test. And I could live with that if there were compile time checking for the bindings in the views. So at least the build would break if you change a property that the view binds to. But it doesn't. That has led us to use GUI automation in our acceptance tests, which is also a bitch to write. But at least it gives us confidence that the system works.
–
PeteJan 30 '11 at 8:52

I like the number (70%). As my teams get higher, I tend to start finding tests for coverage than for value. On the WPF comment, we are just in the beginning days. Meaning, we aren't building/structuring WPF code to be easily testable. Mock models help. When designing code, design it to be testable. And, yes, at this point there are limited examples so you'll have think about it. No different than where most developers were as TDD was introduced to them for the first time just less industry experience.
–
Jim RushJan 30 '11 at 15:25

to be more specific WPF is a bitch to unit test, if you need greater coverage for some reason the easiest way to up coverage of WPF classes is with coded UI tests/integration tests
–
jk.Jan 26 '12 at 8:17

I'm of the opinion that code coverage alone is a poor metric. It's easy to produce tons of useless tests that cover the code, but don't adequately check the output, or don't test edge cases, for example. Covering code just means it doesn't throw an exception, not that it's right. You need quality tests- the quantity isn't that important.

Code Coverage should be one of the outputs of your automated tests that are performed in your automated build system. Tests that don't check the output are hardly worth having. Below-threshold coverage indicates new features with no/insufficient tests. It shouldn't be something to beat people up with - but certainly it can flag up untested code.
–
JBRWilkinsonOct 19 '10 at 21:32

3

I second JBRWilkinson here, although not an indicator of good code, code coverage can be an indicator of a lack of tests. Your unit tests can also deliver other metrics by the way, like performance measures, so that you are not suddenly surprised when a new release comes crashing down the server under the unexpected workload.
–
Matthieu M.Jan 30 '11 at 16:52

4

I think this is a false choice. High-quality tests that only touch a small amount of code are poor overall measures of quality, as are "tests" that touch a large amount of code but don't really check results. Put another way, imagine a quality assurance process for cars that was very thorough at testing the front driver's-side wheel, but no other part of the car. That would be bad in the same way as a QA process that was just a guy eyeballing the whole car and saying "yeah, looks good."
–
Noah RichardsJan 31 '11 at 21:05

"Enough" is when you can make changes to your code with confidence that you're not breaking anything. On some projects, that might be 10%, on others, it might be 95%.

It's almost never as high as 100%. However, sometimes trying to get 100% code coverage can be a great way to remove cruft from the code base. Don't forget that there's two ways to increase code coverage - write more tests or take out code. If code isn't covered because it's hard to test, there's a good chance you can simplify or refactor to make it easier to test. If it's too obscure to bother to test, there's usually a good chance that nothing else in the code is using it.

Only 20% of most code will run 80% of the time. A code coverage analysis is not very useful unless paired with a call graph to determine what needs to be tested the most. That tells you where your edge cases are likely to be. You may come up with 100 tests just for those edge cases, which constitute less than 5% of the actual code.

So, make sure to cover 100% of the 20% that defines critical paths, and at least 50% of the rest (according to the call graph). This should get you (roughly) 70% - 75% total coverage, but it varies.

Don't burn up time trying to get over 70% total coverage while leaving critical edge cases without checks.

At first it makes no sense. Everybody knows that a full test coverage doesn't mean that the code is fully tested and that it is not that difficult to get 100% coverage without actually testing the application.

Nevertheless, 100% coverage is a lower limit: although 100% coverage is not a proof of a bug-free software, it is certain that with a lesser coverage the code is not fully tested and this is simply unacceptable for space critical software.

Use coverage as a guide to indicate areas not tested. Rather than having a mandate for coverage it is wiser to understand the reason for code not covered. Recording a reason for the shortfall is good discipline that allows the risks to be balanced.

Sometimes the reason is less than desirable 'e.g. ran out of time' but might be OK for an early release. It is better to flag areas to return to for a boost in coverage later.

I work on critical flight software where 100% statement coverage is considered suitable for non-critical systems. For the more critical systems we check branch/decision coverage and use a technique call MC/DC which is sometimes not stringent enough.

We also have to ensure that we have also covered the object code.

It is a balance between risk, in our case very high, against value/cost.
An informed choice is needed based upon the risk of missing a bug.

I think it depends on the part of the application you are testing. E.g. for business logic or any component involving complex data transformations, I would aim at 90% (as high as possible) coverage. I have often found small but dangerous bugs by just testing as much of the code as possible. I would rather find such bugs during testing than letting them occur at a customer's site one year later. Also, a benefit of a high code coverage is that it prevents people from changing working code too easily, since the tests have to be adapted correspondingly.

On the other hand, I think there are components for which code coverage is less suited. For example, when testing a GUI it is very time consuming to write a test that covers all the code that gets executed when clicking on a button in order to dispatch the event to the right components. I think in this case it is much more effective to use the traditional approach of performing a manual test in which you just click on the button and observe the behaviour of the program (does the right dialog window open up? does the right tool get selected?).

I don't have that high opinion about using code coverage as a measure for knowing when your test suite has enough coverage.

The main reason why is because if you have a process where you first write some code, then some tests, and then look at the code coverage to discover where you have missed a test, then it is you process that needs improving. If you do true TDD, then you have a code coverage 100% out of the box (admittedly, there are some trivialities I don't test for). But if you look at the code coverage to find out what to test, then you will likely write the wrong tests.

So the only thing you can conclude from the code coverage is that if it is too low, you don't have enough tests. But if it is high, there is no guarantee that you have all the right tests.