Introduction

Writing tests is a difficult and time-consuming activity, and yet
it is a crucial part of good software engineering. Randoop
automatically generates unit tests for Java classes.
Randoop has had many
successful applications, especially with library classes (such as
java.util). Randoop is used at companies like ABB and Microsoft, and
on open-source
projects.

Here
is a JUnit test case
generated by Randoop that reveals an error in OpenJDK (comments
added by hand):

// This test shows that the JDK collection classes
// can create an object that is not equal to itself.
@Test
public static void test1() {
LinkedList list = new LinkedList();
Object o1 = new Object();
list.addFirst(o1);
// A TreeSet is an ordered collection. According to the API
// documentation, this constructor call should throw a
// ClassCastException because the list element is not Comparable. But
// the constructor silently (and problematically) accepts the list.
TreeSet t1 = new TreeSet(list);
Set s1 = Collections.synchronizedSet(t1);
// At this point, we have successfully created a set (s1)
// that violations reflexivity of equality: it is not equal
// to itself! This assertion fails at run time on OpenJDK.
org.junit.Assert.assertTrue(s1.equals(s1));
}

Randoop outputs two kinds of tests:

error-revealing tests that detect bugs in your current code, and

regression tests that can be used to detect future bugs.

You should fix the errors
revealed by the error-revealing tests suite then rerun Randoop, until it
generates no more error-revealing tests. Then, you can run the regression
tests whenever you
change your source code, to be notified of any changes you make to the
behavior of your program. You can always re-run Randoop to check for new
errors, to produce tests for newly-written code, or to regenerate tests
after a code change that causes desirable behavior-changes.

Randoop's tests are not designed to be short or easy to read —
but you will only read them on the rare
occasions when they fail and reveal a bug or a regression failure.

Installing Randoop

Download and unzip the file
randoop-3.1.5.zip.
Set an environment variable
RANDOOP_PATH to
the path of the unzipped archive,
and set an environment variable
RANDOOP_JAR to
the location of randoop-all-3.1.5.jar
within the unzipped archive.

Alternately, if you want to view or work with the source code, follow the instructions in
the Getting Started
section of the Randoop Developer's Manual.

Running Randoop

Run Randoop by invoking its main class randoop.main.Main:

java -ea randoop.main.Main commandargs...

The classpath needs to contain both Randoop
(randoop-all-3.1.5.jar) and the
classes under test.
For instance, if the current working directory contains a directory named
bin containing your compiled classes, then you would use:

... -classpath bin:$(RANDOOP_JAR) ...

(On Windows, adjust the classpath, such as using semicolon instead of colon
as the separator.)
Note that you don't need the -classpath argument if you add
$(RANDOOP_JAR) to your classpath.

Randoop supports two commands:

help prints out a usage message.
For details, see Getting Help.
Example uses:

Example: Generating tests for java.util.Collections

Randoop's tests only use
classes you specify. In order
to effectively test Collections, you should probably also specify some
helper classes, including classes that generate collections. For this
example, we will add java.util.TreeSet to the mix.

Create a file myclasses.txt that lists the names of the classes under test:

After 60 seconds, Randoop stops generating tests. The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following:

Created file: my/home/directory/RegressionTest0.java
Created file: my/home/directory/RegressionTest.java
Created file: my/home/directory/ErrorTest0.java
Created file: my/home/directory/ErrorTest.java
done.

Randoop creates two different main test suites, in classes
ErrorTest and RegressionTest. Compile and run the tests.
(The classpath should include the code under test, the generated tests, and
JUnit files junit.jar and hamcrest-core.jar.)

All of the following are accessible from the package of the tests
(set with --junit-package-name):
M's class, M, and M's
return type and parameter types.

A method or constructor M that satisfies the above conditions is called
a "method under test".
Randoop only calls methods under test; that is, these are the only
methods that will appear in a test case.

The methods under test must include not only the methods whose results you
wish to appear in an assert statement, but also methods that a
test may need to call in order to set up state or create argument values.
All of those methods are also being implicitly tested by the
assert statement.

Use raw type names such as List, not parameterized types such
as List<String>, when specifying classes or methods.

Do not include methods that side-effect global state.
(Methods that side-effect the state of their receiver or other arguments are fine.)
The reason is that such methods create dependent tests that cannot be run
in isolation or reordered.
For example, if test A side-effects global state and test B reads global
state, then whether test B fails depends on whether test A is run before
test B.

WARNING: If you ask Randoop to test code that modifies your file
system (such as File.delete()),
then Randoop will generate tests that modify your file system!
Be careful when choosing
classes and methods to test.

Classifying tests

Randoop generates many tests internally, and it classifies each generated test as
error-revealing, expected behavior, or invalid. The classification depends
primarily on whether the last statement throws an exception or
violates a contract.

You can use command-line arguments to customize Randoop's rules for classifying tests.

Checked exceptions: By default, Randoop classifies a test that throws
a checked exception as expected behavior. You can override this default
with the --checked-exception
command-line option; specify
--checked-exception=error,
--checked-exception=expected, or
--checked-exception=invalid.

The default behavior for
--unchecked-exception is to
treat throwing any unchecked exception as expected, normal behavior. An
alternative would be to consider a test that throws
AssertionError
as error-revealing. However, a random test generator often supplies illegal
arguments (such as null to a method that requires a non-null
argument), and this approach will lead to false alarms in which Randoop
outputs an "error-revealing test" that misuses the software under test.
Depending on your needs, you can override Randoop's default.

A thrown exception is not considered a contract violation unless the
exception is
tagged as error behavior. This is because a method may have a precondition that
is unknown to Randoop.
For example, a binary search implementation might
require that its input is sorted. Randoop should not mark the method
as buggy just because it throws an exception when Randoop passes it an
ill-formed unsorted array. Randoop has no way of knowing each method's
precondition, so by default it assumes that any exception thrown by a
method is correct behavior in response to the values that it was passed.
You can customize how exceptions are considered by
assigning the exception behavior type.

You can also provide specifications of your code to further help
Randoop classify tests; see the checkRep
mechanism.

Error-revealing tests

When Randoop includes a error-revealing test in file ErrorTest.java,
the test shows that the code
violates its specification or contract — some property that it is supposed to preserve. For example, the test shown in
the Introduction
creates a TreeSet that
violates reflexivity of equality: namely, for every
object o, o.equals(o) should
return true.
The test reveals an error in the TreeSet constructor, which does
not properly check its arguments (and possibly a second error in the
TreeSet.equals method, which should always check if its argument
is the same object as itself and return true in such a case).

The test in the Introduction is nicely
succinct: it shows a small number of method calls leading up to the
assertion violation. Because Randoop's generation is random, the actual
test revealing the error can be significantly longer, and contain many
irrelevant calls that are not necessary to reveal the contract violation.
You may wish to minimize the test case to ease debugging.

What kinds of errors does Randoop check for?

When Randoop calls a method that creates an object, Randoop verifies that
the object is well-formed.
Currently, Randoop checks for the following contracts:

Violation of any of these contracts is highly likely to indicate an error.

You can add additional contracts to Randoop.

Regression tests

The tests in the RegressionTest*.java files record the current behavior
(values returned or exceptions thrown) of the classes under test. These
tests assert that behavior, and they currently pass. After you change your
code, you can run these regression tests, which will alert you if your
code changes affect the external behavior of the classes.

This test would fail if you introduced an error that caused contains to return true on an empty set.

A regression test can also assert what exception is currently thrown,
and it will fail if the code is changed to no longer throw that exception.

Regression test failures

If a regression test fails, there are three possible causes. You need to
debug the test failure to determine which is the cause, and then take the
appropriate action.

You have introduced an error — your code changes are incorrect.
In this case, you should correct your mistake and re-run the test suite.

You have intentionally changed the visible behavior of your program. The
regression test has corroborated the change.
In this case, you need to update the tests. Alternately, you can just
discard them and generate new ones.

The tests are too sensitive, and they have failed even though you made
an inconsequential change. For example, maybe
the tests depend on the order of elements in a hash
table.
In this case, you need to re-run Randoop, informing it not to depend on
observations that may vary from run to run. You can use the
--omitmethods command-line
option, or you can replace its behavior with a different implementation
(such as a deterministic one or a mock) using the map_calls feature of the Randoop agent.
You can also see the techniques suggested in the paper
Scaling Up Automated Test Generation.

Usually, a change to your program causes many test failures, not just one.
This is because Randoop generates many tests for each part of your
program's functionality. Don't panic! In most cases, all of the failures
have the same cause.

You may wish to verify the cause of the failures. A way to do this is to
change the part of the program, that you suspect is the cause, back to its
previous behavior. Once you are satisfied that you understand the
failures, you can discard the tests and have Randoop generate new ones.
For example, suppose that you believe that all the test failures are caused
by a change to method m. You can change m back to its
previous implementation and run the tests. If they all succeed, you can
change m back to its new implementation and re-generate the tests.
(As an alternative to making a temporary change to your source code, you
can use
the map_calls feature of the Randoop
agent to replace the new functionality by the old functionality.)

Customizing Randoop's behavior to your application

If you run Randoop without any command-line arguments or customizations to
your environment, it will produce helpful tests. However, Randoop will
produce much more helpful tests if you spend a little bit of time to tell
Randoop about your application, and that is how we recommend using Randoop.

Command-line options

Code under test: which methods may be called by a test

--testclass=string[+].
The fully-qualified name of a class to test; for example, --testclass=java.util.TreeSet
. All of its methods are methods under test. This class is tested in addition to any
specified using --classlist, and must be accessible from the package of the tests
(set with --junit-package-name).

--classlist=filename.
File that lists classes to test. All of their methods are methods under test.

In the file, each class under test is specified by its fully-qualified name on a separate
line. See an
example. These classes are tested in addition to any specified using --testclass
. All classes must be accessible from the package of the tests (set with
--junit-package-name).

--methodlist=filename.
File that lists methods to test.

In the file, each each method under test is specified on a separate line. The list of
methods given by this argument augment any methods determined via the --testclass
or --classlist option.

A constructor line begins with "cons :" followed by the classname, the string
<init>, and the constructor's parameter types enclosed in parentheses. Methods are
specified in a similar way. For example:

--omitmethods=regex.
Randoop will not attempt to directly call methods whose java.lang.reflect.Method.toString() matches the regular expression given. This does not
prevent indirect calls to such methods from other, allowed methods.

Randoop only calls methods that are specified by one of the --testclass,
-classlist, or --methodlist command-line options; the purpose of
--omitmethods is to override one of those other command-line options.

--omit-field-list=filename.
File that contains fully-qualified field names to be excluded from test generation. Otherwise,
Randoop includes all public fields of classes under test as observer methods.

--only-test-public-members=boolean.
Restrict tests to only include public members of classes. Ordinarily, the setting of
--junit-package-name and package accessibility is used to determine which members will
be used in tests. Using this option restricts the tests to only use public members even if the
class is a member of the same package as the generated tests. [default false]

--silently-ignore-bad-class-names=boolean.
Ignore class names specified by user that cannot be found [default false]

--include-if-classname-appears=regex.
Classes, one of which every test must use. Randoop will only output tests whose source code has
at least one use of a member of a class whose name matches the regular expression.

--include-if-class-exercised=filename.
File containing fully qualified names of classes that the tests must exercise. This option only
works if Randoop is run using the exercised-class
javaagent to instrument the classes. A test is output only if it exercises at least one of
the class names in the file. A test exercises a class if it executes any constructor or method
of the class, directly or indirectly (the constructor or method might not appear in the source
code of the test). Included classes may be abstract.

--ignore-flaky-tests=boolean.
If false, Randoop halts and gives diagnostics about flaky tests -- tests that behave
differently on different executions. If true, Randoop ignores them and does not output them.

Use of this option is a last resort. Flaky tests are usually due to calling Randoop on
side-effecting or nondeterministic methods, and a better solution is not to call Randoop on
such methods. [default false]

Which tests to output

--no-error-revealing-tests=boolean.
Whether to output error-revealing tests. Disables all output when used with
--no-regression-tests. Restricting output can result in long runs if the default values
of --inputlimit and --timelimit are used. [default false]

--no-regression-tests=boolean.
Whether to output regression tests. Disables all output when used with
--no-error-revealing-tests. Restricting output can result in long runs if the default
values of --inputlimit and --timelimit are used. [default false]

--no-regression-assertions=boolean.
Whether to include assertions in regression tests. If false, then the regression tests contain
no assertions (except that if the test throws an exception, it should continue to throw an
exception of the same type). Tests without assertions can be used to exercise the code, but
they do not enforce any particular behavior, such as values returned. [default false]

--check-compilable=boolean.
Whether to check that generated sequences can be compiled. If true, the code for each generated
sequence is compiled, and the sequence is only kept if the compilation succeeds without error.
This check is useful because the assumptions in Randoop generation heuristics are sometimes
violated by input methods, and, as a result, a generated test may not compile. This check does
increases the runtime by approximately 50%. [default true]

Test classification

--checked-exception=enum.
If a test throws a checked exception, should it be included in the error-revealing test suite
(value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value:
INVALID)? [default EXPECTED]

ERROR Occurrence of exception reveals an error

EXPECTED Occurrence of exception is expected behavior

INVALID Occurrence of exception indicates an invalid test

--unchecked-exception=enum.
If a test throws an unchecked exception other than OutOfMemoryError,
StackOverflowError, and NullPointerException, should the test be included
in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or
should it be discarded (value: INVALID)?

--npe-on-null-input=enum.
If a test that passes null as an argument throws a NullPointerException
, should the test be be included in the error-revealing test suite (value: ERROR),
regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default EXPECTED]

ERROR Occurrence of exception reveals an error

EXPECTED Occurrence of exception is expected behavior

INVALID Occurrence of exception indicates an invalid test

--npe-on-non-null-input=enum.
If a test that never passes null as an argument throws a
NullPointerException, should the test be included in the error-revealing test suite
(value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value:
INVALID)? [default ERROR]

ERROR Occurrence of exception reveals an error

EXPECTED Occurrence of exception is expected behavior

INVALID Occurrence of exception indicates an invalid test

--oom-exception=enum.
If a test throws an OutOfMemoryError exception, should it be included in the
error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should
it be discarded (value: INVALID)? [default INVALID]

ERROR Occurrence of exception reveals an error

EXPECTED Occurrence of exception is expected behavior

INVALID Occurrence of exception indicates an invalid test

--sof-exception=enum.
If a test throws a StackOverflowError exception, should it be included in the
error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should
it be discarded (value: INVALID)? [default INVALID]

ERROR Occurrence of exception reveals an error

EXPECTED Occurrence of exception is expected behavior

INVALID Occurrence of exception indicates an invalid test

--toradocu-conditions=filename[+].
Read Toradocu JSON condition file to use Toradocu generated conditions to control how tests are
classified.

Param-conditions are used as pre-conditions on method/constructor calls, with test sequences
where the condition fails being classified as BehaviorType.INVALID.

Throws-conditions are used to check exceptions: if the inputs to the call satisfy the
condition, when the exception is thrown the sequence is BehaviorType.EXPECTED, but, if
it is not, the sequence is classified as BehaviorType.ERROR. If the throws-condition is
not satisfied by the input, then ordinary classification is applied.

Test generation stops when either the time limit (--timelimit) is reached, OR the number of
generated sequences reaches the input limit (--inputlimit), OR the number of error-revealing
and regression tests reaches the output limit (--outputlimit).

The default value is appropriate for generating tests for a single class in the context of a
larger program, but is too small to be effective for generating tests for an entire program.

Note that if you use this option, Randoop is nondeterministic: it may generate different
test suites on different runs. [default 100]

--outputlimit=int.
The maximum number of regression and error-revealing tests to output. Test generation stops
when either the time limit (--timelimit) is reached, OR the number of generated sequences
reaches the input limit (--inputlimit), OR the number of error-revealing and regression tests
reaches the output limit (--outputlimit).

In the current implementation, the number of tests in the output can be substantially
smaller than this limit.

If there is no output, this limit has no effect. There is no output when using either
--dont-output-tests or --no-error-revealing-tests together with
--no-regression-tests. [default 100000000]

--inputlimit=int.
Maximum number of test method candidates generated internally. Test generation stops when
either the time limit (--timelimit) is reached, OR the number of generated sequences reaches
the input limit (--inputlimit), OR the number of error-revealing and regression tests reaches
the output limit (--outputlimit). The number of tests output will be smaller than then number
of test candidates generated, because redundant and illegal tests will be discarded. [default 100000000]

--maxsize=int.
Do not generate tests with more than this many statements. [default 100]

--stop-on-error-test=boolean.
Stop generation once an error-revealing test has been generated. [default false]

Values used in tests

--null-ratio=double.
Use null with the given frequency as an argument to method calls.

For example, a null ratio of 0.05 directs Randoop to use null as an input 5
percent of the time when a non-null value of the appropriate type is available.

Unless --forbid_null is true, a null value will still be used if no other value
can be passed as an argument even if --null-ratio=0.

Randoop never uses null for receiver values. [default 0.05]

--forbid-null=boolean.
Do not use null as input to methods or constructors, even when no other argument
value can be generated.

If true, Randoop will not generate a test when unable to find a non-null value of
appropriate type as an input. This could result in certain class members being untested. [default false]

--literals-file=string[+].
A file containing literal values to be used as inputs to methods under test, or "CLASSES".

Literals in these files are used in addition to all other constants in the pool. For the
format of this file, see documentation in class randoop.reflection.LiteralFileReader.
The special value "CLASSES" (with no quotes) means to read literals from all classes under
test.

--literals-level=enum.
How to use literal values that are specified via the --literals-file command-line
option. See: ClassLiteralsMode. [default CLASS]

NONE do not use literals specified in a literals file

CLASS a literal for a given class is used as input only to methods of that class

PACKAGE a literal is used as input to methods of any classes in the same package

ALL each literal is used as input to any method under test

--string-maxlen=int.
Maximum length of strings in generated tests, including in assertions. Strings longer than 65KB
(or about 10,000 characters) may be rejected by the Java compiler, according to the Java
Virtual Machine specification. [default 10000]

Varying the nature of generated tests

--alias-ratio=double.
Try to reuse values from a sequence with the given frequency. If an alias ratio is given, it
should be between 0 and 1.

A ratio of 0 results in tests where each value created within a test input is typically used
at most once as an argument in a method call. A ratio of 1 tries to maximize the number of
times values are used as inputs to parameters within a test. [default 0.0]

--small-tests=boolean.
Favor shorter sequences when assembling new sequences out of old ones.

Randoop generates new tests by combining old previously-generated tests. If this option is
given, tests with fewer calls are given greater weight during its random selection. This has
the overall effect of producing smaller JUnit tests. [default false]

--clear=int.
Clear the component set each time it contains the given number of inputs.

Randoop stores previously-generated tests in a "component" set, and uses them to generate
new tests. Setting this variable to a small number can sometimes result in a greater variety of
tests generated during a single run. [default 100000000]

Outputting the JUnit tests

--testsperfile=int.
Maximum number of tests to write to each JUnit file [default 500]

--junit-package-name=string.
Name of the package for the generated JUnit files. When the package is the same as the package
of a class under test, then package visibility rules are used to determine whether to include
the class or class members in a test. Tests can be restricted to public members only by using
the option --only-test-public-members. [default ]

--junit-before-each=string.
Name of file containing code text to be added to the @Before-annotated method of each generated test class. Code is uninterpreted, and, so, is not run
during generation. Intended for use when run-time behavior of classes under test requires setup
behavior that is not needed for execution by reflection. (The annotation @Before
is JUnit 4, and @BeforeEach is JUnit 5.)

--junit-after-each=string.
Name of file containing code text to be added to the @After-annotated method of each generated test class. Intended for use when run-time behavior of
classes under test requires tear-down behavior that is not needed for execution by reflection.
Code is uninterpreted, and, so, is not run during generation. (The annotation @After
is JUnit 4, and @AfterEach is JUnit 5.)

--junit-before-all=string.
Name of file containing code text to be added to the @BeforeClass
-annotated method of each generated test class. Intended for use when run-time
behavior of classes under test requires setup behavior that is not needed for execution by
reflection. Code is uninterpreted, and, so, is not run during generation. (The annotation
@BeforeClass is JUnit 4, and @BeforeAll is JUnit 5.)

--junit-after-all=string.
Name of file containing code text to be added to the @AfterClass-annotated method of each generated test class. Intended for use when run-time behavior of
classes under test requires tear-down behavior that is not needed for execution by reflection.
Code is uninterpreted, and, so, is not run during generation. (The annotation @AfterClass
is JUnit 4, and @AfterAll is JUnit 5.)

--junit-output-dir=string.
Name of the directory to which JUnit files should be written

--dont-output-tests=boolean.
Run test generation without output. May be desirable when running with a visitor.

NOTE: Because there is no output, the value of --outputlimit will never be met,
so be sure to set --inputlimit or --timelimit to a reasonable value
when using this option. [default false]

--junit-reflection-allowed=boolean.
Whether to use JUnit's standard reflective mechanisms for invoking tests. JUnit's reflective
invocations can interfere with code instrumentation, such as by the DynComp tool. If that is a
problem, then set this to false and Randoop will output tests that use direct method calls
instead of reflection. The tests will include a main method and will execute
methods and assertions, but won't be JUnit suites. [default true]

Runtime environment

-D--system-props=string[+].
Specify system properties to be set (similar to java -Dx=y)

--agent=string.
Specify an extra command for recursive JVM calls that Randoop spawns. The argument to the
--agent option is the entire extra JVM command. A typical invocation of Randoop might be:

--capture-output=boolean.
Capture all output to stdout and stderr [default false]

Controlling randomness

--randomseed=int.
The random seed to use in the generation process. Note that Randoop is deterministic: running
it twice will produce the same test suite, so long as the program under test is deterministic.
If you want to produce multiple different test suites, run Randoop multiple times with a
different random seed. [default 0]

--log=filename.
Name of a file to which to log lots of information. If not specified, no logging is done.

Advanced extension points

--visitor=string[+].
Install the given runtime visitor. See class randoop.ExecutionVisitor.

Threading and timeouts

--usethreads=boolean.
If true, Randoop executes each test in a separate thread and kills tests that take too long to
finish. Tests killed in this manner are not reported to the user.

Use this option if Randoop does not terminate is usually due to execution of code under test
that results in an infinite loop. The downside of this option is a BIG (order-of-magnitude)
decrease in generation speed. The tests are not run in parallel, merely in isolation. [default true]

--timeout=int.
After this many milliseconds, a non-returning method call, and its associated test, are stopped
forcefully. Only meaningful if --usethreads is also specified. [default 5000]

[+] marked option can be specified multiple times

Avoiding calls to specific methods

The code under test may make calls to certain methods that you do not want
Randoop to execute. Two examples are System.exit (which will
terminate not just the target program but also Randoop!) and methods that
require user interaction, such as popping up a modal dialog box.

You can prevent Randoop from calling certain methods in the code under
test, by using the
--omitmethods=regexp command-line option.

Alternately, you can provide a different implementation of a method that will be used
when running under Randoop. Randoop transforms the code under test,
replacing each call to a undesired method by a call to a new implementation
that you provide.

To transform calls to undesired methods, use the Java agent in the
mapcall-3.1.5.jar
file (either in the unzipped distribution file, or downloaded from the
latest release).
Then use this file pass the following command-line option to java when running Randoop:

(Note: If you supply the -javaagent option to java
when running Randoop, and there is any chance that Randoop may spawn
another process (for example, if the --compare-checks command-line
option is specified), then then you should also specify the
--agent argument to Randoop.)

You need to supply the map_calls.txt file, and also a definition
of the replacement methods. When you run the generated tests, you will
also need to supply the same -javaagent:... argument. (In the
future, Randoop may provide a way to generate standalone tests even when
generating tests using the -javaagent:... argument.)

The map_calls.txt file consists of sections, where each section
has the format

Each section gives a regular expression that matches class names, and one or
more replacements that will be done in classes that match the regexp. The
replacement is the fully-qualified method name, fully-qualified argument
types, and a new class name. A call to the old method will be replaced by
a call to a method with identical name and arguments in the new class.

Determining which calls to place in the map_calls file

If Randoop is calling methods that it should not (for example, while
generating tests, a dialog box pops up), then you need to add more calls to
you map_calls file. While the dialog box is displayed, determine which
calls are executing by examining the stack trace. On Unix this can be done
by typing Ctrl-\ (hold down the Control key, then press the
Backslash key).

To determine the specific method, examine the application's code where it
calls the method that puts up the box (or other unwanted activity). If it
is a virtual call, the map must specify the name of the type used in the
call.

Specifying representation invariant methods (such as checkRep)

The "representation invariant" of a class is a property that all
instances of the class must satisfy. For example, a field might
contain a non-negative number, or two fields might contain lists of
the same length.
Sometimes, a program contains a method that checks the rep invariant,
for debugging purposes.
If the rep invariant is ever violated, the program has a bug.

By using the
@CheckRep
annotation (don't forget to import randoop.*), you can
tell Randoop which methods in your classes under test are rep
invariant methods. Randoop will call these methods; if the method
ever fails, Randoop outputs the test as a failing test.

A method annotated with @CheckRep must have one of two
allowed signatures:

A public instance method with no arguments and return
type boolean. In this case, Randoop will interpret a return
value of true as the rep invariant being satisfied, and a
return value of falseor an exception escaping the
method as the rep invariant being violated.

A public instance method with no arguments and return
type void. In this case, Randoop will interpret a normal return
(no exceptions) as the rep invariant being satisfied, and an
exception escaping the method as the rep invariant being violated.

Instrumenting classes for filtering tests on exercised-classes

Randoop will ordinarily generate tests involving all accessible constructors,
fields and methods of tests given using the --testclass and
--classlist options.
These can be restricted to tests that exercise the constructors and methods of
classes from a different list given by using the
--include-if-class-exercised option.
However, for this option to work, the classes must be instrumented to indicate
when they are exercised.
To instrument classes for this filter, also use the exercised-class-3.1.5.jar
by giving the following command-line option for java

-javaagent:$(RANDOOP_PATH)/exercised-class-3.1.5.jar

This will instrument any non-abstract constructor and method of loaded
classes so that the use of a class can be detected.
(The instrumentation is not done on any JDK or JUnit classes.)
(The exercised-class-3.1.5.jar file can be found in the
unzipped distribution archive, or can be downloaded from the
latest release.)

Specifying additional primitive values

By default, Randoop uses the following pool of primitive values as
inputs to methods:

byte: -1, 01, 10, 100

short: -1, 01, 10, 100

int: -1, 01, 10, 100

long: -1, 01, 10, 100

float: -1, 01, 10, 100

double: -1, 01, 10, 100

char: '#', ' ', '4', 'a'

java.lang.String: "", "hi!"

This is the initial set of seed values used by Randoop: in
addition to the above values, any primitive value returned at runtime
from a method call under test may be later used as input to another
method call.

There are two ways of specifying additional seed values to Randoop.

Command line. An easy way to specify additional seed
values in to use the --literals-file command-line option to
either use literals from a file you specify or to read literals out of
the source code that Randoop is analyzing.

Programmatically. You can declare a class that stores
primitive values, using the @TestValue annotation:

Create a public class C, and import randoop.*.

Declare one more more fields in C that hold the
values you want to seed. Each field should:

include the @TestValue annotation,

be declared public and static,

be of a primitive (or String) type, or an array of primitives or Strings, and

be initialized to contain one or more seed values.

Include the class you created in the list of classes given to Randoop to test.

Alternatively, you can simply add a @TestValue annotation
to a field in one of your classes under test, as long as it adheres to
the above requirements.

Below is an example helper class that demonstrates the use of the
@TestValue annotation. For Randoop to use the primitive values
described in the class, you would add this class to the list of
classes under test
(e.g. via the option --testclass=example.TestValueExamples).

Speeding up test generation

To increase the number of tests that Randoop generates in a given amount of
time, run Randoop with --usethreads=false. This will cause
Randoop to
use a single thread to execute all tests, and can speed up the tool by
an order of magnitude.

When running with --usethreads=false (the default), Randoop
executes each generated test in a separate thread. This enables Randoop to
detect and discard non-terminating tests. If you
set --usethreads to false and Randoop appears to
hang, it may be due to a randomly-generated test that is taking too long to
execute and cannot be killed by Randoop. Also see the section
on Randoop non-termination.

Getting help

You can get help on Randoop in the following ways:

Manual: what you are currently reading.

Randoop's help command: For help on
usage, use Randoop's help command. To print Randoop's main help message, run:

java -classpath $(RANDOOP_JAR) randoop.main.Main help

To print help for test generation (the gentests command), run:

java -ea -classpath $(RANDOOP_JAR) randoop.main.Main help gentests

To also include unpublicized (due to being experimental or unsupported)
options for a command, use the --unpub flag:

Bug reporting: If
you have discovered a problem with Randoop, we would like to fix it.
Please submit an issue to the
issue tracker.
You should include enough information to
enable the Randoop developers to reproduce the problem, which will make
it easier for them to diagnose the problem, fix Randoop, and verify the
fix. This generally includes

the exact commands you ran (we recommend using the -ea option to java when running Randoop, which makes diagnosis easier),

the exact and complete output of the commands, and

all the files that are necessary to run those commands.

If you encountered the problem while using the Maven plug-in,
then please reproduce the problem from the command line, to help the
Randoop developers find the underlying bug (or to isolate the problem to the plug-in).

Troubleshooting

Randoop does not run

If the command to run Randoop has the output

Cannot find or load randoop.main.Main

then Randoop is not properly included in the classpath.
One possibility is that you have included the Randoop jar file in the path
specified by the CLASSPATH variable, but have given the location of
the classes-under-test with the -classpath option at the command line.
In this case, the java command will only use the classpath from
-classpath.

It is recommended that both Randoop and the classes under test be included as
the argument to the -classpath option to the java command.
It only makes sense to set CLASSPATH to run Randoop if the location
of classes-under-test will be the same each time you run it.
See the Java Tutorial on
path and classpaths.

If the command to run Randoop has the output

Cannot find the Java compiler. Check that classpath includes tools.jar

Randoop cannot find class-under-test

If when Randoop is run, something like the following is output

No class found for type name "example.ExampleClass"

then the classes under test are not in the classpath.
The most likely case is that you have not included the path to these classes.
Otherwise, the classpath is not given properly.
In particular, if you are running Randoop on Windows, parts of the classpath should be
separated by a semi-colon ";" instead of the colon ":" shown in the examples
in this document.
See the Java Tutorial on
paths and classpaths
for platform specific information on paths.

Randoop does not create tests

If Randoop outputs

No tests were created. No JUnit class created.

then you need to change Randoop's parameters and re-run it. You might give
Randoop more classes to analyze, a larger timeout, additional primitive
values, specific methods to avoid calling, or some other configuration
change. See elsewhere in this manual for suggestions.

Randoop does not create enough tests

Randoop works much better if you provide information to it via
command-line arguments. Spending just a little bit of effort may have a
big impact on Randoop's performance. Find a test that you think Randoop
should be able to generate, and then provide Randoop with the needed
information, or consult with the Randoop developers.

If you suspect the problem may be a bug in Randoop, use the
--log=filename command-line option, and either examine the
log or send it to the Randoop developers for diagnosis.

Randoop produces different output on different runs

Randoop is deterministic: if you run it twice with the same arguments, it
will produce the identical test suite. You can use the
--randomseed command-line
argument to make Randoop produce different tests.

If running Randoop twice produces different test suites, then there is
nondeterminism in your program. Some ways to eliminate nondeterminism in a
sequential Java program are:

Override hashCode(), so that your program does not use
Object.hashCode() which is nondeterministic (it depends on when the
garbage collector runs). Also don't instantiate the Object class,
as in new Object(); instead, use some class that overrides
hashCode().

Mitigate other sources of nondeterminism, such as always calling
Arrays.sort on the result of File.listFiles.

Don't depend on wall-clock or CPU time, either in your program or in
Randoop (for example, by using the
--timelimit command-line
option).

Randoop stopped because of a flaky test

Randoop extends previously-generated tests by adding new method calls to them.
If, in the new longer test, one of the embedded subtests throws an exception
(even though when previously run on its own the subtest passed),
then we say the test is flaky.

Flaky tests are usually due to global side effects, such as setting a
static field; a Randoop-generated test might set the field directly or call
a method that sets the field.
Such methods are not appropriate for randomly-generated
tests, and so you should not call Randoop on such methods. Calling Randoop
on a method that side-effects local state, such as a field of the receiver
or of an argument, is fine.

If Randoop
encounters a flaky test, Randoop halts so that you can correct your
invocation of Randoop, excluding the global-side-effecting method from
consideration. You can do so by passing the
--omitmethods command-line
option.

To help you determine which method to omit, when logging is turned on
(using the --log option) Randoop
logs the list of method calls and field manipulations since the
first execution of the subsequence where the exception occurred.
You should determine which of these methods affected global state, and then
exclude it by using the
--omitmethods command-line
option.

If you find a flaky test that does not result from a method that side-effects
global state, then please help us improve Randoop by submitting an issue via the
issue
tracker to let us know what you encountered.
(Randoop already discards any test with a flaky occurrence of an
OutOfMemoryError or StackOverflowError as invalid.
Additional information can help us find other heuristics that avoid generating
flaky tests, or improve this documentation.)

If you are unable to determine which method caused the flaky tests and
exclude it using the
--omitmethods command-line
option, then you can resort to the command-line argument
--ignore-flaky-tests
to discard the flaky tests.
This is a last resort because it is better to avoid generating flaky tests
rather than discarding them after generating them.

Tests behave differently in isolation or when reordered

Another symptom of calling Randoop on methods that have global side effects, in
addition to flaky tests, is
dependent
tests: tests that behave differently depending on whether some other
test has already run. For instance, test B might succeed if run after test
A, but fail if run on its own.

You can either always run Randoop's generated tests in the same order
(without performing test selection, prioritization, or parallelization),
or you can avoid calling Randoop on methods with global side effects as
described in the section on flaky tests.

Another possible symptom is that if you run Randoop's tests and then your
original test suite, the original test suite might fail because of side
effects performed by Randoop's tests.
To avoid this problem, you can run Randoop's tests in a separate JVM, or
run Randoop's tests last, or avoid calling Randoop on methods with global
side effects as described in the section on flaky
tests.