Jumble is a class level mutation testing tool that works in conjunction with
JUnit. The purpose of mutation testing is to provide a measure of
the effectiveness of test cases. A single mutation
is performed on the code to be tested, the corresponding test cases
are then executed. If the modified code fails the tests, then this
increases confidence in the tests. Conversely, if the modified
code passes the tests this indicates a testing deficiency.

Example

Here is some example Jumble output for a Java class
called "Foo", which has some JUnit tests in a class called "FooTest".

Jumble starts by running the unit tests (in FooTest.class) on the
unmodified Foo class to check that they all pass, and to measure the
time taken by each test. Then it will mutate Foo in various ways and run
the tests again to see if they detect the mutation. It continues this
process until all mutations of Foo have been tried. The output might look
like this:

This says that Jumble has tried 12 different mutants of Foo and the
unit tests (in FooTest) correctly detected the changed behaviour in
8/12 cases (indicated by a '.'), but failed to detect the change in
the other 4/12 cases. For example, when an
if (C) decision on line 31 was mutated to
if (!C), the unit tests still passed, so they
are probably not testing that if statement thoroughly enough.

Overall, 67% of the mutations were detected by
the unit tests, which means that they probably need to be improved.

Key to Symbols

The web interface to Jumble displays the following symbols
as visual feedback of how thoroughly each class is tested.

X, no corresponding test class was found.

No corresponding test class was found, but class possibly contains SWING/AWT code which is hard to JUnit test.

F, very lazy programmer, JUnit code is worthless.

E

D-

D

C-

C

B

A-

A, aim for this as a minimum in production code.

A+

A++, all mutations were detected.

Class was abstract or class did not contain any possible mutation points.

Jumble History

Jumble was developed in 2003-2006 by a commercial company in New
Zealand, Reel Two (www.reeltwo.com), and is now available as open
source under the GPL licence.

JUnit [1] has become the de facto unit testing framework for the
Java language. A class and its corresponding JUnit test is a sensible
granularity at which to apply mutation testing. With Java it is feasible
to perform mutations at either the source code or byte-code level. Jester
[2] is a mutation tester operating at the source code level. While Jester
proves useful, it is hampered by the costly cycle of modifying the source,
compiling the source, and running the tests.

Jumble is a new mutation tester operating directly on class files. It
uses the byte-code engineering library (BCEL) [3] to directly modify class
files thereby drastically cutting the time taken for each mutation test
cycle.

Jumble has been designed to operate in an industrial setting with large
projects. Heuristics have been included to speed the checking of
mutations, for example, noting which test fails for each mutation and
running this first in subsequent mutation checks. Significant effort has
been put into ensuring that it can test code which runs in environments
such as the Apache webserver. This requires careful attention to class
path handling and co-existence with foreign class-loaders.

At ReelTwo, Jumble is used
on a continuous basis within an agile programming environment with
approximately 400,000 lines of Java code under source control. This
checks out project code every fifteen minutes and runs an incremental
set of unit tests and mutation tests for modified classes.