Practical Unit Testing - a manual

An article on the details and HowTos of Unit Testing on the .NET platform

Introduction

Recently, with the
advent of agile programming methods and especially “Extreme Programming”, a
technique somewhat older has been projected into the public view. This
technique, Unit Testing is the general subject of this article. A particular
aspect of it, Automated Unit Testing, has gained popularity with the release of JUnit, an automated testing tool for Java. Since then, a plethora of books,
articles and papers have been written that have Unit Testing as their subject.
Unfortunately, most of the literature on the topic has been focused on the whys
of Unit Testing and on the tools of automated unit testing demonstrated for
their own sake. These writings generally present Automated Unit Testing merely
as a supporting technique for one or another agile programming method.
Furthermore, beyond the documentation (help files, “cookbooks”) provided with
these tools, code examples are rarely given and real examples of the application
of this powerful technique are few and far between.

This article is the first in a series of articles that aim at
filling the gap that exists, providing a complete introduction to Unit Testing
together with code examples that will try to keep as close as possible to real
life scenarios. Obviously, Unit Testing can be done on many platforms and in
many languages but this article will restrict the scope of investigations to the
.NET platform and will be using C# as its implementation
language.

The meaning of Unit Testing

As can be immediately understood by its name,
Unit Testing is a form of testing. More precisely, it is an attempt to prove the
correctness of one or more pieces of software by putting them to work and
validating their functioning/results against a known set of values/behavioural
elements (the contract). This process can take many forms but the object of this
article will be writing code to test code. Basically, given a piece of code (implementation code) of which the
correctness needs to be established, some more code (test code) is written to exercise the
implementation code and verify that operations performed by it correspond to the
model that is being implemented. One key feature of Unit Testing, in fact the
feature that gives it its name, is that it is all about testing discrete
functional units of the software and not the software system as a whole. This
differentiates it from the aptly named System Testing. A unit test ensures the
validity of a well delimited piece of the system (a unit). In practice, this
will mean that what we test are components, classes, packages, compilation
units… depending on the terminology that applies in the environment for which we
develop. Since our main focus of interest here is .NET which is an
object-orientated programming framework, we will be applying Unit testing to
test classes and components.

What
do you test for?

As briefly stated above, what we test for is the adherence of our
code to a set of pre-determined decisions about its workings. That set of
decisions is generally referred to as the
contract. The contract reflects what we wanted to do when we set out to
write that piece of code. It also states what our users (business customers or
other developers using our libraries) expect from our code. As might be
expected, the contract encompasses many things, all of them important to the
users. But generally, three areas can be isolated as being the target of our
testing efforts.

Accuracy

The general compliance of the results
given by our code with what is exhibited by the model we are trying to
implement. Basically, we are making sure here that given correct inputs/working
conditions, our code will produce the right results/perform the proper
operations. I.e. if our code implements the addition process, it must give the
correct sum of the input elements given to it. Compliance with this part of the
contract determines the correctness of our code.

Stress

This area of the contract states the
behaviour expected of our code when the size of inputs given to it is large to
the point of being near overwhelming or when the resources available to our code
are restricted in some way (not enough memory, no ports available, …) It also
describes the expected behaviour of the code in a multi-threaded environment. If
our code performs image processing operations, it must behave properly (not
crash or hang the machine) when given an image that is larger than the memory
available. Compliance with this part of the contract determines the robustness
of our code.

Failure

A part of the contract that spells
out the behaviour expected of our code when it is presented with invalid or
wrong input. To reuse the image processing code example, we assume that our code
is asked to open a file that does not contain an image but is in fact a renamed
music file; in that case, the code must not crash or hang or worse “open” that
file and display a nonsensical picture to the user.

Compliance with this part of the
contract determines the reliability of our code.

Benefits of Unit Testing

Like all other kinds of testing, Unit Testing finds errors, and
just like other kinds of testing it is limited in its scope because you can not
be sure to have found every single error. But Unit Testing helps in finding a
number of them. A particularly interesting application of Unit Testing is in
regression testing; that helps ensure that previously found (and eliminated)
errors do not creep back into the software. Because of its discrete nature, Unit
Testing is easy to perform. Each piece of software being tested is small
(relative to the whole) and therefore well-determined. Because of that, Unit
Testing gives an incredible amount of confidence in one’s code and enables
developers to work and apply changes to complex areas of software knowing that
unit tests are there to maintain the correctness of their
work.

Basically two “schools of thought” exist and
each has its prescribed way of doing Unit Testing. The one that is most vocal
advocates for Test-Driven Development or “test first programming”. What this
means is that given the contract that the implementation code must fulfil, the
programmer starts by writing code to test the implementation (which hasn’t been
written yet). Following that, the rest of the development consists in making
sure that the tests formerly written all pass. This way of doing things is
mostly associated with the Extreme Programming camp. Other developers prefer to
write the code first and then write unit tests to ensure the validity of their
code. Obviously, each “side” has many arguments to justify its position and the
intensity of the debate around this is nearly as great as the one surrounding
coding style or language preference. In the next installment in this series we
will see a bit of both ways.

A
glance at Unit Testing

Keep a running update of any changes or improvements
you've made here.

Let us take a quick look at what Unit Testing is really about. We
will be using, for the sake of this example, a very simplistic situation. The
intent here is to illustrate the process of writing unit tests. Let us assume
that we have to write some code to divide integer numbers. We intend to do so by
creating a class ArithmeticOperations that will expose a method
Division that
takes two integer parameters and returns an integer, the quotient of the
division of the first parameter by the second. We want to be sure that this code
performs properly by writing some unit tests for it. First, we start by listing
known facts about division. This step is important because these facts are the
yardstick against which we will measure the correctness of our division
code.

Any number divided by 1 yields itself

Any number divided by itself yields 1

19 divided by 7 yields 2

15 divided by 5 yields 3

3 divided by 8 yields 0

No number can be divided by zero

What you can notice here is that “Fact” 1 and “Fact” 2 are general facts about the result of a
division. “Fact” 3, “Fact” 4 and “Fact” 5 each show a special case for a division. They have
been chosen to illustrate the three main situations of a division (and this is
important, unit test code should be
written to exercise all types of situations your code will be confronted with);
i.e. an even division, a division with a remainder and a division that gives 0.
“Fact” 6 shows us a case where division cannot be performed (remember the
failure tests?). It is good to list among the “Facts”, special cases for which
we know the result or behaviour to expect and general rules about the result of
our operations. So armed with these facts, we now write another class to test
ArithmeticOperations (actually we could put our tests in the same class but it
is cleaner to separate them from the implementation
code).

We make sure that we provide inputs that match the conditions of
each “Fact”, and we verify that the result provided or behaviour exhibited by
our class corresponds to what is specified in the
“Fact”.

Notice that our code throws an
exception when zero is passed as a divisor. It is very important that all code
(not only contrived examples) actively reject parameters for
which the operation they are supposed to perform is undefined or impossible to
perform.

“Fact” 1 through “Fact” 5 are accuracy tests while “Fact” 6 is a
failure test. This article does not illustrate any stress test. This is partly
because the example is trivial, and also because stress testing requires a
discussion of its own. This is therefore left for the next installment to
address.

All there is left to do is create a console project that will
reference the subject class and the test class, and call each method of the test
class.

We now have a full battery of tests that we
can run on our original class (i.e. ArithmeticOperations). As long as our code
complies with the “Facts” we listed during the test preparation phase, the tests
will run smoothly (no error message). It should be clear by now that the more
“Facts” we know about the code we are writing, the more comprehensive our
testing will be. In the particular case we are studying some more facts could
have been listed (e.g. when multiplying the quotient by the divisor, one obtains
a number which when subtracted from the dividend gives a number always lower
than the divisor. Goodness that sounds complicated!!) but the case is so trivial
that more tests would have made our study disproportionately complicated (if
that’s not already the case J). A trap to avoid is the listing of “Facts” that include one
another. For example, listing a “Fact” that states a rule about division by 2
and another “Fact” that states a rule about division by even
numbers.

Automated Unit Testing and its
tools

The way of doing things displayed in the preceding paragraph,
though functional, tends to be cumbersome and a bit inefficient.
Specifically:

We need to write some more code to call every
method of our test code.

In our test code, we have to call methods to
alert the user of problems.

We are limited to running our tests in a
console environment

It would take some more code to get statistics
about our tests.

This is where Automated Unit Testing comes into play. What we need
is a tool that would provide us with a framework to write our tests and would
enable us to run those tests. That tool would reduce the amount of
infrastructure code we need to write (all those Console.WriteLine(...)
we had in
our test code and the extra project we have to create in order to run the
tests). That tool would allow us to
write the tests once and run them in multiple environments (console mode,
graphical mode). Finally that tool would provide us with complete statistics
about our tests (how many did we run, how many passed, which ones passed, where
did the ones who failed fail…)

NUnit can be gotten at http://www.nunit.orgg . The latest
version is v2.2 which looks a bit different than the version used in this
article v2.1 but nevertheless implements the same
concepts.

Every method that tests one of the “Facts”
mentioned earlier is called a test
case.A class (like TestArithmeticOperations) which
combines a group of test cases is called a test
fixture

When testing a component (basically many
classes that work together to provide a user with a specific interface to
accomplish a well-defined task), one writes many test fixtures. The
combination of those test fixtures is called a test
suite.

So, NUnit allows the programmer to write test suites, test
fixtures and test cases. The mechanism through which this is done is by putting
assertionss about your operations in
your test code. For example, assuming there is a variable r in your code that, after some
computation, should have the value 4. Test code as we wrote it in the previous
paragraph would state

if (r != 4){Console.WriteLine(“Error”);}

Assertion.AssertEquals(“Error”, 4,
r);That code means, [make sure the value of r is 4; if it is not, fail the test
case with the message “Error”]

There are different types of
assertions:

Check for exp being null Assertion.AssertNull(msg,
exp)

Check for exp not being null Assertion.AssertNotNull(msg,
exp)

Check for exp2 being equal to exp1 Assertion.AssertEquals(msg,
exp1, exp2)

Check for exp2 being the same as exp1
Assertion.AssertSame(msg,
exp1, exp2)

Check for exp being a true boolean expression
Assertion.Assert(msg,
exp)

All assertions make the test case fail if the condition they are
testing for is not verified. In that case they issue the message specified by msg.

A special kind of assertion fails: Assertion.Fail(msg)

Once the methods containing the
assertions have been written, they need to be signalled as being test cases. The
NUnit framework defines a way for the programmer to do just that. All that needs
to be done is to add the attribute [Test]]
to the
method.

TestFixture] attribute to the
class.

No special marker is needed to signal a group of classes as a test
suite.

Once this test code has been written, what is the next step? First
of all, we make sure the whole thing compiles (implementation code and test
code). You may choose to have two separate projects (the best choice for large
scale development) or combine test code and implementation code in the same
project. If you have separate projects, be sure to add a reference to the
implementation project in the test project.

NUnit offers two operating modes Console and GUI modes. Let us
look at the console mode:

The console application nunit-console.exe located in the bin
sub-folder of the NUnit installation folder runs the test fixtures present in
the test assembly with which it is invoked.

nunit-console <test assembly>

Here is the invocation and results of the tests we just wrote on
the ArithmeticOperations class.

As you can see, we had 6 tests run (the number of the “Facts” we
defined), none of them failed; we also had the amount of time it took to run
these tests.nunit-console offers more
sophisticated options to run the tests. Help on those are listed by calling
nunit-console without any option. The other mode offered by NUnit is the GUI
mode.

The Windows Form application nunit-console.exe located in the bin
sub-folder of the NUnit installation folder runs the test fixtures present in
the test assembly with which it is invoked.

nunit-gui <test assembly>

After using that command line, the Main NUnit form is
loaded

Click on the ‘Run’ button and see the
progress bar unroll: Green at the beginning and shifting to Red as soon as a
single test is failed. In our case, the bar remains green all through and we end
up with this

This way of implementing the division
(which we could have chosen for whatever reason) intentionally has bugs which
our tests can help us “discover”.

Running the tests from the command prompt again give us
this:

Running the tests from the Graphical User Interface gives us
this:

In the right window pane, we have a view of all the tests, those
that passed in green, the ones that failed in red. The progress bar is entirely
red to signify that our class/component is not correct. A window below the
progress bar shows us the details of the test failures (same as in the console
mode).

Conclusion

Add-ins to Visual Studio.NET (NUnitAdd-in,
VSNUnit…)

Testing frameworks for ASP.NET
(NUnitAsp)

Test reporting tools, test coverage tools…
(NUnit2Report…)

Even though Unit testing is not the
ultimate all-error remover weapon (studies have shown that it detects only
30%-40% of all bugs), it is a great tool for making sure that those bugs that
have been removed remain out of the software and as such boosts developer
productivity while helping to maintain software quality standards.

Like any other powerful tool, it must be applied with discipline
or it will harm the development process. One particular pitfall to watch for is
the accumulation over time of unit tests making the software builds
prohibitively long and causing developers to waste an incredible amount of time
running tests every time they want to check in new code to the repository (or
worse stopping to run the tests because of the time they take). Special policies
about unit tests and the build process must then be made and followed. These
issues have to do with continuous integration and will be the object of a future
article.

In all, when properly applied, unit testing has benefits that you
as a developer cannot afford to pass.

In the next installment in this
series, we will look at a general method for writing unit tests building upon
the recommendations given in this article and we will see some detailed examples
of real-world development with unit tests.

We will touch upon Test-Driven
Development with an example of such a process and we will delve into stress
testing discussing some of its complexities.

In essence, yes. The method has to have some side effects (changing some object members, writing to a file, opening sockets ...).
If the method does not return any value and does not change anything then it's not doing any work and does not need to be tested

If i want add Junit into c# program how i add it. Can i use C# to generate a testing coding system for checking java coding? i want to know is it got any problem for solving this problem. Any computer professional please help me....

Thanks for the article. I especially like what you said about Contracts.

I am trying to add Unit Testing to my own personal process. I have NUNIT and it is working fine. All of the articles I have read have discussed using a tool like NUNIT and the value of Unit testing. There is also some trivial code to test. I am looking for some more real life kinds of advice. I am starting a new iteration of my current development, and I wanted to add unit testing. I believe in the test first paradigm, but I wanted to add Unit Tests to what I already have. I am interested in finding out how much testing is appropriate and having a standard method to generating the tests. The real goal is to be able to have someone review the code and tests with a checklist that says you have done enough.

When going back over the code, how much unit test is enough. I have heard that you should test every potential path in your code. Most things seem really trivial and I feel very silly writing unit tests for them. Accessors? Mutators? Checking for null parameters? This stuff seems really silly when you already have the code written and you have already manually tested or at least ad-hoc tested. On the other hand, if you do not have a clear guideline for what needs to be tested, you may very well think of something as being "trivial", and find a bug that you missed. Where is the balance.

The next thing is to test classes that are user interface elements. How are they tested efficiently. I have been trying to factor out all I can from the user interface, so my UI classes are basically just event handlers that call something else so I can test them. Same type of thing with classes that are used as callbacks from a framework.

Anyway, I think I am going in the right dorection, but I could use some more guidance I think in these areas.

Typically, I test ALL the public methods in my design. No matter how trivial. Where unit testing really shines is when you are making changes to your code at a later date(refactoring or bug fixes). That's the time when seemingly trivial bits of code break. Having tests for those bits of code helps you detect that.
Testing the user interface is indeed a big problem; but the approach you have chosen is the right one. By keeping the UI as simple possible (getting info from the user and calling events) you will be able to easily write tests that make sure that the right events are called with the right data.
Assuming here that you are using Visual Studio.NET, you probably noticed that the way it leads you to put event code straight into the UI is not the best. It makes your UI very hard to test (not to mention the coupling issues in your code)
A better way would be to use an event manager; I recommend you read Marc Clifton's article on that topic (http://www.codeproject.com/csharp/eventmanager.asp[^]

Stay tuned for the next installment in my unit testing series. (I will get around to putting it together this week).

Thanks for the feedback.
I wanted to really get into testing my app from a very high level, from the user's perspective. I wrote a Mock User object that acts the way we have been describing, with its own little hooks for performing actions. I REALLY liked the idea of testing from the user's perspective. I know it is not a "Unit Test" per se, rather an automated integration test. I feel MUCH more confident that I am testing something of value. I added stubs for doing white box unit testing, but I have found that I can almost always cover everything from a very high level instead.

The first run at it was very ugly. Lots of delegate callbacks and the like. It made it very difficult to follow the tests. I knew that I wanted the test to look just like the steps in the Use Case.

Perform Action 1
Wait for some event
Assert a few things

Perform action 2
Wait for some event
Assert some more things

To do this, I had to write it muti-threaded. The test thread does some actions, sets a static variable that directs the main thread to do something, sleeps and passes control to the main thread. Back and forth between the main thread having control and the test thread having control. It took a bit to get this going, but it is working pretty well. I occasionally get stuck in programmatically trying to get a third party UI control to do something programmatically, but for the most part it is going well.

That sounds very exciting!
I have a question though as to the generality of this code.
How do you customize this code tho work with a different UI?
Is it scriptable, configurable (XML, ...) ?
Or is it locked into the current program you are working with.
You should consider writing an article about your efforts and the results you got...

Thanks, it has been a fun little project. As far as the code being configurable or scriptable, it is not right now, but I can see how it might be. I do have one little section of code that has reusable standard NUNIT like testing stuff. I used NUNIT for a bit, It was pretty and all with the nice green status bar, but I didn't like the rules it was making me stick to. I also had a hard time getting it to let me trace into my own code when a test failed. That stuff is so easy to write that I just made a very simple version for myself that did what I wanted it to do.

The main bulk of the Mock-User stuff is very specific to what I am writing. I am trying to stick to the XP paradigm of write what you need when you need it, rather than making some other framework. I am not worried about making it work with another UI for the moment. This is also my first shot at it, so I wanted to see what really works before making any kinds of generalizations, although the principles should follow for any project. It might be an interesting article.

I usually don't like adding scripting stuff if I can avoid it. Why learn ANOTHER language, even if it is XML. The reflection stuff in .NET is so dang powerful, why not make C# my scripting language? Do you have any insights in this area?