Marty Andrews

artful code

Saturday, February 24, 2007

How many tests do you need?

The number of tests you need for your system depends on how complex it is. Obvious, right? The tricky part is knowing how to measure that. There are a couple of good mathematical definitions that can help.

The Cyclomatic Complexity of a method is the number of linearly independent paths through a program's source code. This means that the Cyclomatic Complexity of a method is the minimum number of unit tests needed to test every path in a method.

Every method starts out with a default Cyclomatic Complexity of one. The complexity is then increased by one for every linearly independent path through that method. In this case, each if statement in the method represents an extra path. So the total Cyclomatic Complexity for the method is four.

In other words, if you were writing tests for this method, you should write at least four of them.

The NPath Complexity of a method is the number of acyclic execution paths through that method. This means that the NPath Complexity basically represents all of the combinations of paths possible through a method, and is an upper bound on the number of tests you need to write for a method.

Lets use the same example code again.

Every method starts out with a default NPath Complexity of one. The complexity is then increased by one for every nested conditional loop, and multiplied by two for every parallel conditional loop. In this case, the method starts out with a complexity of one. It is multiplied by two at the first if block, bringing it up to two. It is multiplied by two again at the second if block, bringing it up to four. It is multiplied by two yet again at the last if block, bringing it up to eight.

In other words, if you were writing tests for this method, you should write up to eight of them.

So the number of tests you need to write is:

Cyclomatic Complexity <= Number of Tests <= NPath Complexity

The hard part now is to try and calculate what your complexity is. Fortunately, Complexian can do that for you.