Why We Refactored JUnitThe Story of an Open Source Endeavorby Bill Venners, with Matt Gerrans and Frank SommersJanuary 27, 2003

Summary
Artima SuiteRunner, a free, open source unit and conformance testing toolkit for
Java, was released this week. Find out why the authors felt the need to create Artima
SuiteRunner and learn how you can use it to run JUnit test suites.

This week Matt Gerrans, Frank Sommers, and I released Artima SuiteRunner, a free
open source testing toolkit for Java. You can use this tool with JUnit to run existing
JUnit test suites, or standalone to create unit and conformance tests for Java APIs. JUnit,
originally written by Erich Gamma and Kent Beck, is very popular among Java
developers and especially the extreme programming (XP) community. Like most Java
developers, when we needed to create tests we first turned to JUnit. But as we started
using JUnit in our own projects, we encountered some limitations. We created a new tool,
Artima SuiteRunner, that better addressed our testing needs. In this article, we explain
why JUnit didn't fit the bill, and how Artima SuiteRunner was created.

Creating a Conformance Test Kit for the ServiceUI API

As leader of the Jini community's ServiceUI project, I wrote a specification and
implemented an API that defines a standard way to attach user interfaces to Jini services.
Within the Jini community, the ServiceUI API quickly became a de facto standard, but I
wanted to make it official. To propose the ServiceUI API as an official Jini community
standard through the Jini Decision Process, however, I needed a conformance test kit
(CTK). To help me build the ServiceUI CTK, I turned to JUnit.

My initial experience with JUnit was both pleasant and frustrating. I quickly learned
how to create a test case and write test methods, but found it difficult to run a test suite.
In fact, I spent three hours sitting with my laptop at a coffee shop in Ft. Collins,
Colorado, trying hard not to pull my hair out in frustration. But once I set up JUnit to run
my test suite, I was quite happy. I liked the simplicity of creating JUnit test cases and
adding test methods. I liked the red or green status bar of JUnit's graphical runner. Except
for one special signature test, I proceeded contentedly to write the entire ServiceUI CTK
with JUnit.

Before long I had only one missing piece remaining in my ServiceUI CTK: a
signature test. A signature test verifies the artifacts of an API's signature,
such as name, type, parameters, access level, and other modifiers. It does not verify the
API's semantics. Although not necessary for unit testing, a signature test is a critical part
of any conformance test. A signature test ensures that each type, field, and method
required by an API specification is actually present in an implementation.

Terence Parr, creator of ANTLR, once said, "Why write something in five days that
you can spend five years automating?" When it was time to write the signature test
portion of the ServiceUI CTK, I decided rather than write it by hand, I would write a
generic tool to automatically generate signature tests. I figured I would design this tool as
an add-on to JUnit, and release it open source, primarily for the benefit of the Jini
community.

So I wrote a tool to generate signature tests, and then attempted to make it work with
JUnit. Once again, my hair was in danger of being pulled, but this time it was worse. My
previous difficulty was primarily in figuring out how to use JUnit as an
application. Now I was trying to figure out how to use JUnit as an
API. At this point I was 9000 feet above sea level, in Crested Butte,
Colorado. Perhaps the thin air affected my cognitive abilities, but at one point I just threw
up my hands and said to myself, "Why is this so hard? It would be easier to rewrite JUnit
than figure it out."

Just some friendly advice: if you ever find yourself saying it would be easier to
rewrite something rather than figure it out, slap yourself. Chances are you are wrong. I
certainly was. Creating SuiteRunner was a huge amount of work. Despite my frustrations
with JUnit, it would have been orders of magnitude easier to decipher JUnit's source code
than to create a brand new testing toolkit.

Nevertheless, I was fairly far along in the process of creating Artima SuiteRunner
before I realized this, and along the way two of my friends, Matt Gerrans and Frank
Sommers, became involved in the project. So despite many moments of doubt, including
one decision to cancel the project entirely, ultimately Matt, Frank, and I decided to finish
what we had started.

A Little Help from My Friends

Initially I got some help from Matt Gerrans, who wanted a Java project to work on to
improve his Java skills. Although a veteran programmer, Matt had spent most of the
second half of the nineties programming in languages such as C++ and Python. Over
many months, Matt and I, in between rounds of foosball, drew circles and arrows on
white boards and hacked away at code. The design and implementation of Artima
SuiteRunner evolved through much discussion and many iterations.

Frank Sommers became involved with Artima SuiteRunner project after running into
limitations using JUnit to test network services. Frank and a group of students from the
University of Southern California had built a set of Jini services that formed the
infrastructure of a cluster-computing framework. They needed to test not only each Jini
service implementation, but also the system as a whole, under a variety of conditions over
a sustained period of time. They started to write JUnit test cases, but soon ran into
problems.

Their problems with JUnit started even before their initial test would execute. In fact,
that was exactly the problem: the first tests would not execute. Jini's infrastructure relies
on Java's ability to load classes into a running Java virtual machine (JVM) from any
network location. Dynamic class loading alleviates the need to pre-install service
implementation code at every Jini client. But Jini's mobile code requires that class
loading at a Jini client conform to the delegation-based class loading mechanism
specified in the JVM 1.2 specifications. Unfortunately, JUnit's graphical runner did not
follow that class loading model, and worked in a way that prevented class loading from
the network URLs annotated into their Jini stubs. That made testing Jini services with
JUnit's graphical runner impossible. Class loading worked correctly in JUnit's text runner,
but that meant having to forgo the graphical UI.

As soon as Frank and his team overcame their initial hurdle by using JUnit's text
runner, they encountered a second obstacle. Since each test result represented a sample of
the overall system reliability, they wanted to collect many such samples in a format
amenable to statistical analysis. They found JUnit's reporting facility simplistic for their
purposes. To achieve their testing objective, they faced a choice: extend JUnit's reporting
capability, or look for another tool more suitable for their testing requirements.

When Frank Sommers heard about the Artima SuiteRunner project, he tried it and
immediately sent Matt and me joyous emails. Even though we didn't know about Frank's
problems at the time, our Reporter notion was perfect for Frank's needs. So Frank joined
the project and contributed much design insight, code, and time.

Adding Value to JUnit

JUnit is the de facto standard open-source unit test toolkit for Java. Countless
organizations have written countless JUnit test cases. We realized most developers would
never use Artima SuiteRunner if it meant throwing out their investment in JUnit test
cases, so we needed to make sure Artima SuiteRunner added value on top of JUnit. We
wanted to find ways to help JUnit users get more out of their existing JUnit test suites.

One difference between JUnit and Artima SuiteRunner is that in JUnit, the notion of
results reporting is linked to the notion of running test suites. For example, to generate
test results as text, you use JUnit's text runner. To generate graphical test results, you use
JUnit's Swing runner. By contrast, Artima SuiteRunner separates the notion of running
from that of reporting. In Artima SuiteRunner, we provide one runner that can be
configured with multiple reporters. Each reporter presents customized test
results: for example, a text reporter presents results as text, a graphical reporter presents
results graphically.

Because JUnit links the concepts of running and reporting in the notion of a runner,
we made Artima SuiteRunner a JUnit runner. You can use Artima SuiteRunner to run and
report results of JUnit tests, just like you can use JUnit's text or Swing runners. The three
main advantages of using SuiteRunner to run your JUnit tests are reporters, runpaths, and
recipe files:

A reporter collects and presents test results in a highly customizable way to
the user. Examples are text, graphics, Web pages, database output, CSV files, XML, and
email alerts.

A runpath lets you load classes for your tests from anywhere with an easy-to-configure list of filenames, directory paths, and/or URLs.

A recipe file captures and saves in a file the run properties of a particular
suite of tests for easy reuse.

Artima SuiteRunner can help developers create unit and conformance tests, whether or
not they have an existing investment in JUnit test cases.

Rethinking JUnit

Originally, I intended to create a tool that generated signature tests for conformance
test kits using JUnit. Frank Sommers originally intended to test the reliability of network
services and the distributed system they composed using JUnit. Both of us ran into
trouble. The beauty of open source is that when a tool doesn't quite fit your needs, you
have the option to morph the code into a new form that better solves your problem. Open
source projects are occasionally forked, and you can consider Artima SuiteRunner to be a
fork of JUnit. But it is a design fork not a code fork, because we didn't reuse any JUnit
code. We rethought and reworked JUnit's ideas, and wrote Artima SuiteRunner's code
from scratch.

Coming Soon to Artima.com: Refactoring Tales

In the coming weeks and months, I will publish a series of "Refactoring Tales"
articles on Artima.com, each of which focuses narrowly on one aspect of the JUnit and
Artima SuiteRunner APIs. Each Refactoring Tales article will in effect be a highly
focused design review of one aspect of both JUnit and Artima SuiteRunner. Through
these articles I hope to stimulate discussion about API design in the Artima Forums. If
you would like to be notified each week of new articles at Artima.com, including the
Refactoring Tales series, please subscribe to the Artima
Newsletter.