zope.testrunner uses buildout. To start, run python bootstrap.py. It will
create a number of directories and the bin/buildout script. Next, run
bin/buildout. It will create a test script for you. Now, run bin/test
to run the zope.testrunner test suite.

LP #719369: An Unexpected success (concept introduced in Python 2.7) is
no longer handled as success but as failure. This is a workaround. The
whole unexpected success concept might be implemented later.

The testrunner module is used to run automated tests defined using the
unittest framework. Its primary feature is that it finds tests by
searching directory trees. It doesn’t require the manual
concatenation of specific test suites. It is highly customizable and
should be usable with any project. In addition to finding and running
tests, it provides the following additional features:

Test filtering using specifications of:

o test packages within a larger tree

o regular expression patterns for test modules

o regular expression patterns for individual tests

Organization of tests into levels and layers

Sometimes, tests take so long to run that you don’t want to run them
on every run of the test runner. Tests can be defined at different
levels. The test runner can be configured to only run tests at a
specific level or below by default. Command-line options can be
used to specify a minimum level to use for a specific run, or to run
all tests. Individual tests or test suites can specify their level
via a ‘level’ attribute. where levels are integers increasing from 1.

Most tests are unit tests. They don’t depend on other facilities, or
set up whatever dependencies they have. For larger applications,
it’s useful to specify common facilities that a large number of
tests share. Making each test set up and and tear down these
facilities is both ineffecient and inconvenient. For this reason,
we’ve introduced the concept of layers, based on the idea of layered
application architectures. Software build for a layer should be
able to depend on the facilities of lower layers already being set
up. For example, Zope defines a component architecture. Much Zope
software depends on that architecture. We should be able to treat
the component architecture as a layer that we set up once and reuse.
Similarly, Zope application software should be able to depend on the
Zope application server without having to set it up in each test.

The test runner introduces test layers, which are objects that can
set up environments for tests within the layers to use. A layer is
set up before running the tests in it. Individual tests or test
suites can define a layer by defining a layer attribute, which is
a test layer.

The test runner consists of an importable module. The test runner is
used by providing scripts that import and invoke the run method from
the module. The testrunner module is controlled via command-line
options. Test scripts supply base and default options by supplying a
list of default command-line options that are processed before the
user-supplied command-line options are provided.

Typically, a test script does 2 things:

Adds the directory containing the zope package to the Python
path.

Calls the test runner with default arguments and arguments supplied
to the script.

Normally, it just passes default/setup arguments. The test runner
uses sys.argv to get the user’s input.

This testrunner_ex subdirectory contains a number of sample packages
with tests. Let’s run the tests found here. First though, we’ll set
up our default options:

we see the normal testrunner output, which summarizes the tests run for
each layer. For each layer, we see what layers had to be torn down or
set up to run the layer and we see the number of tests run, with
results.

The test runner returns a boolean indicating whether there were
errors. In this example, there were no errors, so it returned False.

(Of course, the times shown in these examples are just examples.
Times will vary depending on system speed.)

A Layer is an object providing setup and teardown methods used to setup
and teardown the environment provided by the layer. It may also provide
setup and teardown methods used to reset the environment provided by the
layer between each test.

Layers can extend other layers. Note that they do not explicitly
invoke the setup and teardown methods of other layers - the test runner
does this for us in order to minimize the number of invocations.

Now lets specify a layer in the suite containing TestSpecifyingNoLayer
and run the tests again. This demonstrates the other method of specifying
a layer. This is generally how you specify what layer doctests need.

Now lets also specify a layer in the TestSpecifyingNoLayer class and rerun
the tests. This demonstrates that the most specific layer is used. It also
shows the behavior of nested layers - because TopLayer extends BaseLayer,
both the BaseLayer and TopLayer environments are setup when the
TestSpecifyingNoLayer tests are run.

If we inspect our trace of what methods got called in what order, we can
see that the layer setup and teardown methods only got called once. We can
also see that the layer’s test setup and teardown methods got called for
each test using that layer in the right order.

It is possible to force the layers to run in subprocesses and parallelize them.

>>> sys.argv = [testrunner_script, '-j2']
>>> testrunner.run_internal(defaults)
Running samplelayers.Layer1 tests:
Set up samplelayers.Layer1 in N.NNN seconds.
Ran 9 tests with 0 failures and 0 errors in N.NNN seconds.
Running samplelayers.Layer11 tests:
Running in a subprocess.
Set up samplelayers.Layer1 in N.NNN seconds.
Set up samplelayers.Layer11 in N.NNN seconds.
Ran 26 tests with 0 failures and 0 errors in N.NNN seconds.
Tear down samplelayers.Layer11 in N.NNN seconds.
Tear down samplelayers.Layer1 in N.NNN seconds.
Running samplelayers.Layer111 tests:
Running in a subprocess.
Set up samplelayers.Layerx in N.NNN seconds.
Set up samplelayers.Layer1 in N.NNN seconds.
Set up samplelayers.Layer11 in N.NNN seconds.
Set up samplelayers.Layer111 in N.NNN seconds.
Ran 26 tests with 0 failures and 0 errors in N.NNN seconds.
Tear down samplelayers.Layer111 in N.NNN seconds.
Tear down samplelayers.Layerx in N.NNN seconds.
Tear down samplelayers.Layer11 in N.NNN seconds.
Tear down samplelayers.Layer1 in N.NNN seconds.
Running samplelayers.Layer112 tests:
Running in a subprocess.
Set up samplelayers.Layerx in N.NNN seconds.
Set up samplelayers.Layer1 in N.NNN seconds.
Set up samplelayers.Layer11 in N.NNN seconds.
Set up samplelayers.Layer112 in N.NNN seconds.
Ran 26 tests with 0 failures and 0 errors in N.NNN seconds.
Tear down samplelayers.Layer112 in N.NNN seconds.
Tear down samplelayers.Layerx in N.NNN seconds.
Tear down samplelayers.Layer11 in N.NNN seconds.
Tear down samplelayers.Layer1 in N.NNN seconds.
Running samplelayers.Layer12 tests:
Running in a subprocess.
Set up samplelayers.Layer1 in N.NNN seconds.
Set up samplelayers.Layer12 in N.NNN seconds.
Ran 26 tests with 0 failures and 0 errors in N.NNN seconds.
Tear down samplelayers.Layer12 in N.NNN seconds.
Tear down samplelayers.Layer1 in N.NNN seconds.
Running samplelayers.Layer121 tests:
Running in a subprocess.
Set up samplelayers.Layer1 in N.NNN seconds.
Set up samplelayers.Layer12 in N.NNN seconds.
Set up samplelayers.Layer121 in N.NNN seconds.
Ran 26 tests with 0 failures and 0 errors in N.NNN seconds.
Tear down samplelayers.Layer121 in N.NNN seconds.
Tear down samplelayers.Layer12 in N.NNN seconds.
Tear down samplelayers.Layer1 in N.NNN seconds.
Running samplelayers.Layer122 tests:
Running in a subprocess.
Set up samplelayers.Layer1 in N.NNN seconds.
Set up samplelayers.Layer12 in N.NNN seconds.
Set up samplelayers.Layer122 in N.NNN seconds.
Ran 26 tests with 0 failures and 0 errors in N.NNN seconds.
Tear down samplelayers.Layer122 in N.NNN seconds.
Tear down samplelayers.Layer12 in N.NNN seconds.
Tear down samplelayers.Layer1 in N.NNN seconds.
Running zope.testrunner.layer.UnitTests tests:
Running in a subprocess.
Set up zope.testrunner.layer.UnitTests in N.NNN seconds.
Ran 156 tests with 0 failures and 0 errors in N.NNN seconds.
Tear down zope.testrunner.layer.UnitTests in N.NNN seconds.
Tearing down left over layers:
Tear down samplelayers.Layer1 in N.NNN seconds.
Total: 321 tests, 0 failures, 0 errors in N.NNN seconds.
False

Sometimes, There are tests that you don’t want to run by default.
For example, you might have tests that take a long time. Tests can
have a level attribute. If no level is specified, a level of 1 is
assumed and, by default, only tests at level one are run. to run
tests at a higher level, use the –at-level (-a) option to specify a higher
level. For example, with the following options:

If the –progress (-p) option is used, progress information is printed and
a carriage return (rather than a new-line) is printed between
detail lines. Let’s look at the effect of –progress (-p) at different
levels of verbosity.

Note that, prior to Python 2.4, calling pdb.set_trace caused pdb to
break in the pdb.set_trace function. It was necessary to use ‘next’
or ‘up’ to get to the application code that called pdb.set_trace. In
Python 2.4, pdb.set_trace causes pdb to stop right after the call to
pdb.set_trace.

You can also do post-mortem debugging, using the –post-mortem (-D)
option:

The trace module supports ignoring directories and modules based the test
selection. Only directories selected for testing should report coverage. The
test runner provides a custom implementation of the relevant API.

The TestIgnore class, the class managing the ignoring, is initialized by
passing the command line options. It uses the options to determine the
directories that should be covered.

When running the test runner, modules are sometimes created from text
strings. Those should always be ignored:

>>> ignore.names('/myproject/src/blah/hello.txt', '<string>')
True

To make this check fast, the class implements a cache. In an early
implementation, the result was cached by the module name, which was a problem,
since a lot of modules carry the same name (not the Python dotted name
here!). So just because a module has the same name in an ignored and tested
directory, does not mean it is always ignored:

If we delete the source files, it’s normally a disaster: the test runner
doesn’t believe any test files, or even packages, exist. Note that we pass
--keepbytecode this time, because otherwise the test runner would
delete the compiled Python files too:

Finally, passing --usecompiled asks the test runner to treat .pyc
and .pyo files as adequate replacements for .py files. Note that the
output is the same as when running with .py source above. The absence
of “removing stale bytecode …” messages shows that --usecompiled
also implies --keepbytecode:

When having problems that seem to be caused my memory-management
errors, it can be helpful to adjust Python’s cyclic garbage collector
or to get garbage colection statistics. The –gc option can be used
for this purpose.

If you think you are getting a test failure due to a garbage
collection problem, you can try disabling garbage collection by
using the –gc option with a value of zero.

The –report-refcounts (-r) option can be used with the –repeat (-N)
option to detect and diagnose memory leaks. To use this option, you
must configure Python with the –with-pydebug option. (On Unix, pass
this option to configure and then build Python.)

Each layer is repeated the requested number of times. For each
iteration after the first, the system refcount and change in system
refcount is shown. The system refcount is the total of all refcount in
the system. When a refcount on any object is changed, the system
refcount is changed by the same amount. Tests that don’t leak show
zero changes in systen refcount.

It is instructive to analyze the results in some detail. The test
being run was designed to intentionally leak:

class ClassicLeakable:

def __init__(self):

self.x = ‘x’

class Leakable(object):

def __init__(self):

self.x = ‘x’

leaked = []

class TestSomething(unittest.TestCase):

def testleak(self):

leaked.append((ClassicLeakable(), Leakable(), time.time()))

Let’s go through this by type.

float, leak.ClassicLeakable, leak.Leakable, and tuple

We leak one of these every time. This is to be expected because
we are adding one of these to the list every time.

str

We don’t leak any instances, but we leak 4 references. These are
due to the instance attributes avd values.

dict

We leak 2 of these, one for each ClassicLeakable and Leakable
instance.

classobj

We increase the number of classobj instance references by one each
time because each ClassicLeakable instance has a reference to its
class. This instances increases the references in it’s class,
which increases the total number of references to classic classes
(clasobj instances).

type

For most interations, we increase the number of type references by
one for the same reason we increase the number of clasobj
references by one. The increase of the number of type references
by 3 in the second iteration is puzzling, but illustrates that
this sort of data is often puzzling.

int

The change in the number of int instances and references in this
example is a side effect of the statistics being gathered. Lots
of integers are created to keep the memory statistics used here.

The summary statistics include the sum of the detail refcounts. (Note
that this sum is less than the system refcount. This is because the
detailed analysis doesn’t inspect every object. Not all objects in the
system are returned by sys.getobjects.)

Python packages have __path__ variables that can be manipulated to add
extra directories cntaining software used in the packages. The
testrunner needs to be given extra information about this sort of
situation.

Let’s look at an example. The testrunner-ex-knit-lib directory
is a directory that we want to add to the Python path, but that we
don’t want to search for tests. It has a sample4 package and a
products subpackage. The products subpackage adds the
testrunner-ex-knit-products to it’s __path__. We want to run tests
from the testrunner-ex-knit-products directory. When we import these
tests, we need to import them from the sample4.products package. We
can’t use the –path option to name testrunner-ex-knit-products.
It isn’t enough to add the containing directory to the test path
because then we wouldn’t be able to determine the package name
properly. We might be able to use the –package option to run the
tests from the sample4/products package, but we want to run tests in
testrunner-ex that aren’t in this package.

We can use the –package-path option in this case. The –package-path
option is like the –test-path option in that it defines a path to be
searched for tests without affecting the python path. It differs in
that it supplied a package name that is added a profex when importing
any modules found. The –package-path option takes two arguments, a
package name and file path.

The –path option defines a directory to be searched for tests and a
directory to be added to Python’s search path. The –test-path option
can be used when you want to set a test search path without also
affecting the Python path:

It is an error to specify –report-refcounts (-r) without specifying a
repeat count greater than 1

>>> sys.argv = 'test -r'.split()
>>> testrunner.run_internal(defaults)
You must use the --repeat (-N) option to specify a repeat
count greater than 1 when using the --report_refcounts (-r)
option.
<BLANKLINE>
True

>>> sys.argv = 'test -r -N1'.split()
>>> testrunner.run_internal(defaults)
You must use the --repeat (-N) option to specify a repeat
count greater than 1 when using the --report_refcounts (-r)
option.
<BLANKLINE>
True

This can be a bit confusing, especially when there are enough tests
that they scroll off a screen. Often you just want to see the first
failure. This can be accomplished with the -1 option (for “just show
me the first failed example in a doctest” :)

The –show-secondary-failures option counters -1 (or it’s alias),
causing the second and subsequent errors to be shown. This is useful
if -1 is provided by a test script by inserting it ahead of
command-line options in sys.argv.

If a doctest has large expected and actual output, it can be hard to
see differences when expected and actual output differ. The –ndiff,
–udiff, and –cdiff options can be used to get diff output of various
kinds.

Here, the actual output uses the word “and” rather than the word “an”,
but it’s a bit hard to pick this out. We can use the various diff
outputs to see this better. We could modify the test to ask for diff
output, but it’s easier to use one of the diff options.

The –ndiff option requests a diff using Python’s ndiff utility. This
is the only method that marks differences within lines as well as
across lines. For example, if a line of expected output contains digit
1 where actual output contains letter l, a line is inserted with a
caret marking the mismatching column positions.

If there are errors when importing a test module, these errors are
reported. In order to illustrate a module with a syntax error, we create
one now: this module used to be checked in to the project, but then it was
included in distributions of projects using zope.testrunner too, and distutils
complained about the syntax error when it compiled Python files during
installation of such projects. So first we create a module with bad syntax: