Navigation

Doctesting a function ensures that the function performs as claimed by
its documentation. Testing can be performed using one thread or
multiple threads. After compiling a source version of Sage, doctesting
can be run on the whole Sage library, on all modules under a given
directory, or on a specified module only. For the purposes of this
chapter, suppose we have compiled Sage 6.0 from source and the top
level Sage directory is:

See the section Running Automated Doctests for information on Sage’s
automated testing process. The general syntax for doctesting is as
follows. To doctest a module in the library of a version of Sage, use
this syntax:

where --long is an optional argument (see Optional Arguments
for more options). The version of sage used must match the version
of Sage containing the module we want to doctest. A Sage module can be
either a Python script (with the file extension “.py”) or it can be a
Cython script, in which case it has the file extension “.pyx”.

Say we want to run all tests in the sudoku module
sage/games/sudoku.py. In a terminal window, first we cd to the
top level Sage directory of our local Sage installation. Now we can
start doctesting as demonstrated in the following terminal session:

The numbers output by the test show that testing the sudoku module
takes about four seconds, while testing all specified modules took the
same amount of time; the total time required includes some startup
time for the code that runs the tests. In this case, we only tested
one module so it is not surprising that the total testing time is
approximately the same as the time required to test only that one
module. Notice that the syntax is:

In all of the above terminal sessions, we used a local installation of
Sage to test its own modules. Even if we have a system-wide Sage
installation, using that version to doctest the modules of a local
installation is a recipe for confusion.

If your system Python has the tox package, you can also run the Sage
doctester as follows:

To doctest modules of a Sage installation, from a terminal window we
first cd to the top level directory of that Sage installation,
otherwise known as the SAGE_ROOT of that installation. When we
run tests, we use that particular Sage installation via the syntax
./sage; notice the “dot-forward-slash” at the front of
sage. This is a precaution against confusion that can arise when
our system has multiple Sage installations. For example, the following
syntax is acceptable because we explicitly specify the Sage
installation in the current SAGE_ROOT:

In this case, we received an error because the system-wide Sage
installation is a different (older) version than the one we are
using for Sage development. Make sure you always test the files
with the correct version of Sage.

So far we have used a single thread to doctest a module in the Sage
library. There are hundreds, even thousands of modules in the Sage
library. Testing them all using one thread would take a few
hours. Depending on our hardware, this could take up to six hours or
more. On a multi-core system, parallel doctesting can significantly
reduce the testing time. Unless we also want to use our computer
while doctesting in parallel, we can choose to devote all the cores
of our system for parallel testing.

Let us doctest all modules in a directory, first using a single thread
and then using four threads. For this example, suppose we want to test
all the modules under sage/crypto/. We can use a syntax similar to
that shown above to achieve this:

Notice the time difference between the first set of tests and the
second set, which uses the optional argument --long. Many tests in the
Sage library are flagged with #longtime because these are known to
take a long time to run through. Without using the optional --long
argument, the module sage/crypto/mq/sr.py took about five
seconds. With this optional argument, it required 82 seconds to run
through all tests in that module. Here is a snippet of a function in
the module sage/crypto/mq/sr.py with a doctest that has been flagged
as taking a long time:

deftest_consistency(max_n=2,**kwargs):r""" Test all combinations of ``r``, ``c``, ``e`` and ``n`` in ``(1, 2)`` for consistency of random encryptions and their polynomial systems. `\GF{2}` and `\GF{2^e}` systems are tested. This test takes a while. INPUT: - ``max_n`` -- maximal number of rounds to consider (default: 2) - ``kwargs`` -- are passed to the SR constructor TESTS: The following test called with ``max_n`` = 2 requires a LOT of RAM (much more than 2GB). Since this might cause the doctest to fail on machines with "only" 2GB of RAM, we test ``max_n`` = 1, which has a more reasonable memory usage. :: sage: from sage.crypto.mq.sr import test_consistency sage: test_consistency(1) # long time (80s on sage.math, 2011) True """

The main Sage library resides in the directory
SAGE_ROOT/src/. We can use the syntax described above
to doctest the main library using multiple threads. When doing release
management or patching the main Sage library, a release manager would
parallel test the library using 10 threads with the following command:

Another way is run makeptestlong, which builds Sage (if necessary),
builds the Sage documentation (if necessary), and then runs parallel
doctests. This determines the number of threads by reading the
environment variable MAKE: if it is set to make-j12, then
use 12 threads. If MAKE is not set, then by default it uses
the number of CPU cores (as determined by the Python function
multiprocessing.cpu_count()) with a minimum of 2 and a maximum of 8.

Any of the following commands would also doctest the Sage library or
one of its clones:

make test
make check
make testlong
make ptest
make ptestlong

The differences are:

maketest and makecheck — These two commands run the same
set of tests. First the Sage standard documentation is tested,
i.e. the documentation that resides in

SAGE_ROOT/src/doc/common

SAGE_ROOT/src/doc/en

SAGE_ROOT/src/doc/fr

Finally, the commands doctest the Sage library. For more details on
these command, see the file SAGE_ROOT/Makefile.

maketestlong — This command doctests the standard
documentation:

SAGE_ROOT/src/doc/common

SAGE_ROOT/src/doc/en

SAGE_ROOT/src/doc/fr

and then the Sage library. Doctesting is run with the optional
argument --long. See the file SAGE_ROOT/Makefile for further
details.

makeptest — Similar to the commands maketest and makecheck. However, doctesting is run with the number of threads as
described above for makeptestlong.

makeptestlong — Similar to the command makeptest, but
using the optional argument --long for doctesting.

The underlying command for running these tests is sage-t--all. For
example, makeptestlong executes the command
sage-t-p--all--long--logfile=logs/ptestlong.log. So if you want
to add extra flags when you run these tests, for example --verbose,
you can execute
sage-t-p--all--long--verbose--logfile=path/to/logfile.
Some of the extra testing options are discussed here; run
sage-t-h for a complete list.

You can run doctests from within Sage, which can be useful since you
don’t have to wait for Sage to start. Use the run_doctests
function in the global namespace, passing it either a string or a module:

Ideally, doctests should not take any noticeable amount of time. If
you really need longer-running doctests (anything beyond about one
second) then you should mark them as:

sage: my_long_test() # long time

Even then, long doctests should ideally complete in 5 seconds or
less. We know that you (the author) want to show off the capabilities
of your code, but this is not the place to do so. Long-running tests
will sooner or later hurt our ability to run the testsuite. Really,
doctests should be as fast as possible while providing coverage for
the code.

Use the --long flag to run doctests that have been marked with the
comment #longtime. These tests are normally skipped in order to
reduce the time spent running tests:

To find tests that take longer than the allowed time use the
--warn-long flag. Without any options it will cause tests to
print a warning if they take longer than 1.0 second. Note that this is
a warning, not an error:

You can run tests that require optional packages by using the
--optional flag. Obviously, you need to have installed the
necessary optional packages in order for these tests to succeed. See
http://www.sagemath.org/packages/optional/ in order to download
optional packages.

By default, Sage only runs doctests that are not marked with the optional tag. This is equivalent to running

If you’re testing many files, you can get big speedups by using more
than one thread. To run doctests in parallel use the --nthreads
flag (-p is a shortened version). Pass in the number of threads
you would like to use (by default Sage just uses 1):

To doctest the whole Sage library use the --all flag (-a for
short). In addition to testing the code in Sage’s Python and Cython
files, this command will run the tests defined in Sage’s documentation
as well as testing the Sage notebook:

Sometimes doctests fail (that’s why we run them after all). There are
various flags to help when something goes wrong. If a doctest
produces a Python error, then normally tests continue after reporting
that an error occurred. If you use the flag --debug (-d for
short) then you will drop into an interactive Python debugger whenever
a Python exception occurs. As an example, I modified
sage.schemes.elliptic_curves.constructor to produce an error:

Sometimes an error might be so severe that it causes Sage to segfault
or hang. In such a situation you have a number of options. The
doctest framework will print out the output so far, so that at least
you know what test caused the problem (if you want this output to
appear in real time use the --verbose flag). To have doctests run
under the control of gdb, use the --gdb flag:

Sage also includes valgrind, and you can run doctests under various
valgrind tools to track down memory issues: the relevant flags are
--valgrind (or --memcheck), --massif, --cachegrind and
--omega. See http://wiki.sagemath.org/ValgrindingSage for more details.

Once you’re done fixing whatever problems where revealed by the
doctests, you can rerun just those files that failed their most recent
test by using the --failed flag (-f for short):

The first failure in a file often causes a cascade of others, as
NameErrors arise from variables that weren’t defined and tests fail
because old values of variables are used. To only see the first
failure in each doctest block use the --initial flag (-i for
short).

Sometimes tests fail intermittently. There are two options that allow
you to run tests repeatedly in an attempt to search for Heisenbugs.
The flag --global-iterations takes an integer and runs the whole
set of tests that many times serially:

If you are working on some files in the Sage library it can be
convenient to test only the files that have changed. To do so use the
--new flag, which tests files that have been modified or added
since the last commit:

When testing a file which is not part of a package (which is not in a
directory containing an __init__.py file), the testing
code loads the globals from that file into the namespace before
running tests. To disable this behaviour (and require imports to be
explicitly specified), use the --force-lib option.

To give a json file storing the timings for each file, use the
--stats_path flag. These statistics are used in sorting files so
that slower tests are run first (and thus multiple processes are
utilized most efficiently):