Summary
No matter how you arrive at a design, be it UML modeling, TDD micro-steps, or applying patterns, you should take a step back and look at the public API as a whole from a usability perspective. How easy is the API as a whole to learn, understand and use?

Martin Fowler points out some really weird behavior in JUnit. Apparently JUnit creates a new test object for each test. I understand why it does that, and I agree with the principles behind it. But this has some other implications:

If this is the case why the hell bother with a setUp() method? Why not just use the constructor? tearDown() makes a tad more sense, but not much; and I've never actually needed to use it in practice.

If something's going to take a long time to set up, you should make it a local variable in every test that uses it, rather than a field, as long as some tests don't use it.

Discover JUnit's semantics

I discovered that JUnit creates a new TestCase instance to run each test method a few years ago while looking through JUnit's source code. I was reading JUnit's code in an attempt to figure out how to automatically generate API signature tests for JUnit while creating a conformance test kit (CTK) for the ServiceUI API. A signature test verifies the artifacts of an API's signature, such as name, type, parameters, access level, and other modifiers, but does not verify the API's semantics.

When I discovered JUnit was creating a new TestCase instance for each test method, I was suprised. I hadn't understood that by reading the documentation, and it seemed very non-intuitive to me. The documentation emphasized that I should use setUp() to create my fixture to avoid side effects, which to me implied that JUnit was creating one instance of each TestCase, and invoking all test methods on that serially. And as Elliotte Rusty Harold put it, if you're going to create a new TestCase instance for each test method, "why the hell bother with a setUp() method?" You can just use the TestCase constructor.

I've heard Bruce Eckel point out that there is one subtle difference between creating your fixture in setUp() versus creating it in the TestCase constructor. JUnit creates all the TestCase instances up front, and then for each instance, calls setup(), the test method, and tearDown(). In other words, the subtle difference is that constructors are all invoked in batch up front, whereas the setUp() method is called right before each test method. But this seems to be not that useful a difference in practice.

What does "test" mean?

My discovery that JUnit created a new TestCase instance for each test method did help me get some insight into one other aspect of JUnit's API that had confused me: what does "test" mean in the API? Is a test a test method? One would think so, but then what's a TestCase? A TestCase seemed to be a suite of tests that all use the same fixture, so a TestCase is a suite of related tests. Hmm, "TestCase" seems like an odd name for a suite of tests. But wait, TestCase implements Test, which is also implemented by TestSuite. So a TestCase is a Test, and a TestSuite is a Test, and a TestCase is a suite of tests.

When I discovered by looking at the code that JUnit's runner was instantiating a new instance of each TestCase for each test method, I realized that a TestCase did actually map to a test at runtime. Now I finally thought I could perhaps understand where the name TestCase had come from. A TestCase instance was one test and one fixture. Unfortunately, this meant that in JUnit's API, the TestCase class represents one concept (a suite of tests), whereas a TestCase instance represents a different concept (a test). I found that to be very non-intuitive.

When I threw up my hands and wrote SuiteRunner, I collapsed Test, TestCase, and TestSuite into one class called Suite. A Suite represents a conceptual suite of tests. A test method is a kind of test, but not necessarily the only kind. Instead of using the composite pattern as JUnit does with Test, TestCase, and TestSuite, SuiteRunner just uses plain old composition. A Suite can hold references to sub-Suites. This approach requires fewer types in the public API, and uses type names that are consistent with underlying concepts. And that makes the API easier to understand and use.

Ask a Suite to execute itself

Another thing that bothered me about JUnit's design is that the code that decides how to execute a suite of tests is in the runner. It seemed more intuitive and object-oriented to me to ask a suite to execute its own tests. That way the API could provide a default way to execute tests, which the users could override at their option. In JUnit's design, if you want to execute tests differently, you must write a new runner.

In SuiteRunner, therefore, class Suite has an execute() method. Suite's implementation of execute() uses reflection to discover any test methods on itself, invokes them one after another, and invokes execute() on any sub-Suites. If you prefer to create a separate instance for each test method a la JUnit, you could override execute() in a Suite subclass and do just that. In fact, you can do anything you want in execute(). For example, SuiteRunner has a package access Suite subclass called JUnitSuite, whose execute() method uses reflection and dynamic proxies to run a suite of JUnit tests. That's how SuiteRunner can act as a JUnit runner to run JUnit tests.

Conclusion: Design for usability

I originally wrote SuiteRunner to solve a specific problem, a CTK for ServiceUI. One reason I did the extra work (with some help from my friends) to make SuiteRunner a general tool was to demonstrate something I think is important to good design: usability. No matter how you arrive at a design, be it UML modeling, TDD micro-steps, or "applying patterns, one after another, until you have the architecture of the system," you should take a step back and look at the public API as a whole from a usability perspective. Are there ways to minimize the number of types and methods in the public interface? Are types and methods named consistently with each other and with the concepts they represent? Is the behavior intuitive? How easy is the API as a whole to learn, understand and use?

I was mystified by the name "TestCase" too until I figured it out. The word "TestCase" describes the object that is created, not the text of the class.

Each TestCase, at run time, is configured to run exactly one test method so it really is a test case. The fact that subclasses of TestCase can each have many test methods doesn't make "TestCase" a suite. Individual test cases don't ever use all those methods.

The name describes the object, just like the name "Account" is intended to describe each "Account" object. Seeing TestCases as suites is kind of like wanting to call an "Account" class "Accounts".

Interestingly, if the same instance is reused, the name "TestCase" doesn't seem as accurate any more.

> The name [TestCase] describes the object, just like the name> "Account" is intended to describe each "Account" object.> Seeing TestCases as suites is kind of like wanting to> call an "Account" class "Accounts".> I disagree. In just about every other class I've seen, the concept of the class is the same as the concept of the instance. In class Account, for example, each instance represents one account concept, and the class represents the blueprint for one account. In class TestCase, each instance represents one test, but the class represents the blueprint for many tests. That's what makes TestCase an unusual design for me--not "wrong," just unusual, and therefore surprising and non-intuitive. The design tradeoff with that choice is that you get the isolation that prevents side affects, but you make the API a bit harder to learn and understand. It is a small dissonance, but most JUnit users have probably had to scratch their head about it at some point.

> That's what makes TestCase an unusual> design for me--not "wrong," just unusual, and therefore> surprising and non-intuitive. The design tradeoff with> that choice is that you get the isolation that prevents> side affects, but you make the API a bit harder to learn> and understand. It is a small dissonance, but most JUnit> users have probably had to scratch their head about it at> some point.

NUnit 1.0 followed the same design and was just as confusing (to me).

When NUnit 2.0 switched to using attributes instead of inheritance, they made things clearer by requiring a [TestFixture] attribute on classes that previously derived from TestCase.

I can't say if they create a test fixture new instance for each test method defined in the class or not but that seems like an implementation detail that I don't need to concern myself with.

When it comes to using JUnit, I think of the TestCase class like this:

A TestCase derivative with only one test method in it defines a simple test case. A TestCase derivative with multiple test methods in it defines a test fixture.

> I can't say if [NUnit] creates a test fixture new instance for> each test method defined in the class or not but that> seems like an implementation detail that I don't need to> concern myself with.> According to Martin Fowler, NUnit doesn't create a new instance for each test method:

...The popular NUnit implementation doesn't do it. The following NUnit test causes a failure.