Complementing Unit Testing with Static Analysis, with NUnit as an Example

When discussing static analysis tools for C# projects, programmers will often deny the necessity of static analysis arguing that most errors can be caught through unit testing. So, I decided to find out how well one of the most popular unit-testing frameworks, NUnit, was tested and see if our analyzer could find anything of interest there.

Introduction

NUnit is a popular unit-testing library for .NET projects ported from Java to C#. Its code is open and can be downloaded from the project website http://www.nunit.org/.

It should be noted that JUnit — the project that NUnit was ported from — was created by such renowned programmers as Erich Gamma, a co-author of the textbook on object-oriented design patterns, and Kent Beck, the creator of the test-driven development and extreme programming methodologies. I recall reading his book Test Driven Development By Example once, where he explains test-driven development by the example of creating a unit-testing framework, like JUnit, following all of his methodologies. What I mean to say is that there is no doubt that JUnit and NUnit were developed in keeping with the best traditions of unit testing, which is also confirmed by Kent Beck’s comment at the NUnit site: “… an excellent example of idiomatic design. Most folks who port xUnit just transliterate the Smalltalk or Java version. That’s what we did with NUnit at first, too. This new version is NUnit as it would have been done had it been done in C# to begin with.”

I looked through NUnit’s source files: there are piles of tests; it looks like they have tested everything that could be tested. Taking into account the project’s great design and the fact that NUnit has been used by thousands of developers over a number of years, I didn’t expect PVS-Studio to find a single bug there. Well, I was mistaken: it did find one bug.

About the bug found

It triggered the V3093diagnostic, which deals with an issue when programmers use operators & and | instead of && and ||. This issue may cause troubles when it is critical that the right part of an expression should not execute under certain conditions. Let’s see what this error looks like in NUnit.

Even if the Matches method receives the value null as the actual parameter, the right operand of the & operator will be evaluated anyway, which means that the Canonicalize method will be called, too. If you look at this method’s definition, you’ll see that the value of its path parameter is not tested for null and method Replace is called on it right away — this is where NullReferenceException might be raised. Let’s try to reproduce this issue. For that purpose, I wrote a simple unit-test:

Here it is: NUnit crashed with NullReferenceException. PVS-Studio managed to find a real bug even in such a well-tested product as NUnit is. Note that it was no harder than writing a unit-test: you just run project analysis from the menu and check the grid with the results.

Conclusion

Unit testing and static analysis are not alternative, but complementary software-development strategies [1]. DownloadPVS-Studio analyzer and run it on your projects to see if it can find errors that tests didn’t.

References

Comments

Most of us work with strings one way or another. There’s no way to avoid them — when writing code, you’re doomed to concatinate strings every day, split them into parts and access certain characters by index. We are used to the fact that strings are fixed-length arrays of characters, which leads to certain limitations when working with them. For instance, we cannot quickly concatenate two strings. To do this, we will at first need to allocate the required amount of memory, and then copy there the data from the concatenated strings.