Previous topic

Next topic

This Page

Quick search

Matplotlib has a testing infrastructure based on nose, making it easy
to write new tests. The tests are in matplotlib.tests, and
customizations to the nose testing infrastructure are in
matplotlib.testing. (There is other old testing cruft around,
please ignore it while we consolidate our testing to these locations.)

Writing an image based test is only slightly more difficult than a
simple test. The main consideration is that you must specify the
“baseline”, or expected, images in the
image_comparison() decorator. For
example, this test generates a single image and automatically tests
it:

The first time this test is run, there will be no baseline image to
compare against, so the test will fail. Copy the output images (in
this case result_images/test_category/spines_axes_positions.*) to
the correct subdirectory of baseline_images tree in the source
directory (in this case
lib/matplotlib/tests/baseline_images/test_category). Note carefully
the * at the end: this will copy only the images we need to include
in the git repository. The files ending in _pdf.png and
_svg.png are converted from the pdf and svg originals on the fly
and do not need to be in the respository. Put these new files under
source code revision control (with gitadd). When rerunning the
tests, they should now pass.

There are two optional keyword arguments to the image_comparison
decorator:

extensions: If you only wish to test some of the image formats
(rather than the default png, svg and pdf formats), pass a
list of the extensions to test.

tol: This is the image matching tolerance, the default 1e-3.
If some variation is expected in the image between runs, this
value may be adjusted.

If you’re writing a test, you may mark it as a known failing test with
the knownfailureif()
decorator. This allows the test to be added to the test suite and run
on the buildbots without causing undue alarm. For example, although
the following test will fail, it is an expected failure:

We try to keep the tests categorized by the primary module they are
testing. For example, the tests related to the mathtext.py module
are in test_mathtext.py.

Let’s say you’ve added a new module named whizbang.py and you want
to add tests for it in matplotlib.tests.test_whizbang. To add
this module to the list of default tests, append its name to
default_test_modules in lib/matplotlib/__init__.py.

Travis is configured to receive notifications of new commits to GitHub
repos (via GitHub “service hooks”) and to run builds or tests when it
sees these new commits. It looks for a YAML file called
.travis.yml in the root of the repository to see how to test the
project.

If you want to enable Travis CI for your personal matplotlib GitHub
repo, simply enable the repo to use Travis CI in either the Travis CI
UI or the GitHub UI (Admin | Service Hooks). For details, see the
Travis CI Getting Started page. This
generally isn’t necessary, since any pull request submitted against
the main matplotlib repository will be tested.

Testing all versions of Python (2.6, 2.7, 3.*) requires
having multiple versions of Python installed on your system and on the
PATH. Depending on your operating system, you may want to use your
package manager (such as apt-get, yum or MacPorts) to do this.

tox makes it easy to determine if your working copy introduced any
regressions before submitting a pull request. Here’s how to use it:

$ pip install tox
$ tox

You can also run tox on a subset of environments:

$ tox -e py26,py27

Tox processes everything serially so it can take a long time to test
several environments. To speed it up, you might try using a new,
parallelized version of tox called detox. Give this a try:

$ pip install -U -i http://pypi.testrun.org detox
$ detox

Tox is configured using a file called tox.ini. You may need to
edit this file if you want to add new environments to test (e.g.,
py33) or if you want to tweak the dependencies or the way the
tests are run. For more info on the tox.ini file, see the Tox
Configuration Specification.