Managing Your Project's Virtualenvs with Tox

A comprehensive beginner’s introduction to Tox.

Tox is a great tool for standardising and automating any development task that
benefits from being run in an isolated virtualenv. But Tox suffers from an
educational problem: Tox appears to be just for running tests, and that’s all that
many projects use it for. It can also be unclear how best to use the tox.ini
file format at first. This tutorial aims to clear up the confusion and teach
you how to automate a project’s development tasks with Tox. It covers
everything you need to know to understand the real hypothesis/h tox.ini file.

Why Tox is confusing at first

Tox can appear to be just a tool for running tests, rather than for development
tasks in general:

Tox was originally for running your tests in an isolated virtual
environment, but has grown into a more general project virtualenv /
command management tool.

For many developers their first introduction to Tox will be when they come
across it being used in some project to run the tests, while the project
confusingly uses manually-created virtualenvs for other things that Tox
could be used for, like building the documentation and running the
development server.

Tox’s own documentation is unclear on the scope of Tox too, stating Tox’s vision as
to automate and standardize testing in Python. But the same docs later describe Tox
as a generic virtualenv management and test command line tool and
include examples of using Tox to build documentation and to run development environments.
Tox’s GitHub project’s description is Command line driven CI frontend
and development task automation tool.

Internal terminology used by Tox also suggests that it’s just for testing:
Tox refers to the virtualenvs that it creates as testenvs.
We’ll be using both testenv and virtualenv in this guide too.
Eventually we’ll get around to explaining exactly what a testenv is
relative to a virtualenv.

In addition:

The tox.ini file format is just complicated enough that it requires some
knowledge of how Tox works to understand how to write a good one.

The relationship between Tox and similar tools such as Docker, GNU Make,
virtualenv, virtualenvwrapper, etc is unclear.

So what is Tox and what is it for?

Tox is a project virtualenv management and development task automation tool.
You create a tox.ini file for your project, and in this file you define
virtualenvs (Tox calls them testenvs) for all your project tasks that
you want to standardize and automate with Tox: running the tests, running the
development server, running the linters, building the documentation, publishing
a release, etc. Here’s an example tox.ini file that’ll be explained in detail
later:

You run Tox telling it which of your testenvs to run, e.g. tox -e docs to
build the documentation or tox -e tests,lint to run the tests followed by
the linter, and Tox creates a virtualenv, installs the
dependencies that you’ve listed in tox.ini into it, and runs the commands
that you’ve listed in tox.ini in it.

Tox’s workflow

More concretely, Tox is a tool that automates a certain workflow, as described in
the System overview in the
Tox docs:

Create a Python virtual environment using the version of Python selected in tox.ini.

Install any Python dependencies listed in tox.ini (the deps setting) into the virtualenv.
If an sdist of your project was created in step 2 install that too.

Run the commands listed in tox.ini (the commands setting) in the virtualenv.
By default fail if any of the commands exit with a non-zero exit code.

Print out a report of the virtualenvs that were run and whether each succeeded or failed.

So Tox is really just a tool for creating virtualenvs and running commands in them, and any development task that can be
automated by running commands in virtualenvs can be automated with Tox.

If you ask Tox to run more than one virtualenv at once it will loop repeating steps 3, 4 and 5 for each virtualenv, and finally report on them all in step 6.

If you run the same virtualenv again Tox will speed things up by reusing the previously created virtualenv. It’ll skip
steps 3 and 4 and avoid unnecessarily recreating the virtualenv or reinstalling the dependencies unless you pass the
tox --recreate option or unless Tox knows that the virtualenv needs to be recreated (for example because the deps in
tox.ini have changed).

Benefits of using Tox

By defining all of your project’s tasks in a tox.ini file Tox simplifies things
for developers. A developer only needs to install and run Tox, they don’t need
to bother with creating and activating virtualenvs and installing dependencies
themselves.

Tox also simplifies CI integration: CI scripts that just run Tox are much
simpler than scripts that handle virtualenvs and dependencies themselves.

Tox also standardizes things, reducing differences between different
development environments and between dev envs and CI and production.
The tox.ini file defines exactly what dependencies to install into each
virtualenv and Tox takes extra steps to isolate its virtualenvs from the
outside system.

Tox can also reduce the chance of dependency conflicts by using a separate
virtualenv for each task, rather than installing everything in one big
virtualenv. The packages needed to build your documentation or run your linters
don’t need to be installed in the virtualenv that runs your dev server.

Finally, Tox provides one place to define and document all of the
available projects tasks – in the tox.ini file – with a simple and
consistent command line interface for running those tasks.

I call Tox project virtualenv management because it uses a per-project
configuration file, the tox.ini file, that you add to your project’s version
control repository. This is different than other tools, such as
virtualenvwrapper, that are more personal-use virtualenv managers. Your text
editor is your personal text editing tool and text editor configuration or
standardizing on one particular text editor isn’t usually part of a project.
GNU Make on the other hand is a project build automation tool because its
Makefile is tracked as part of the project. Tox is a project tool, like Make.

Tox’s original use-case

The original purpose of Tox was to run tests in isolated environments and (later) to
automate running tests with multiple combinations of different versions of
Python and different versions of different dependencies. It uses some more
recent Tox features, but this example partial tox.ini file for
Compressing a dependency matrix
from Tox’s docs illustrates Tox’s originally intended use-case:

[tox]envlist=py{27,34,36}-django{15,16}-{sqlite,mysql}[testenv]deps=django15:Django>=1.5,<1.6django16:Django>=1.6,<1.7py34-mysql:PyMySQL; use if both py34 and mysql are in an env name
py27,py36:urllib3; use if any of py36 or py27 are in an env name
py{27,36}-sqlite:mock; mocking sqlite in python 2.x

Running the command tox with this tox.ini file will generate the “cross product”
of the different vetsions of Python, Django and SQLite or MySQL and run the project’s
tests many times over: with Python 2.7, Django 15, and SQLite, then with Python 2.7,
Django 15 and MySQL, then with Python 2.7, Django 16, and SQLite, and so on,
for all combinations. More on this “generative envlist” feature
below. The deps setting defines how the dependencies to be
installed should change depending on the versions of Python and Django being
tested and on SQLite or MySQL.

We don’t have a need at Hypothesis to test our code against many different
versions like this. We’re writing a web application that gets deployed to a
production environment that we control, with pinned versions of Python and all
dependencies. We use Tox instead for its isolation and task automation features.

Where we’re going: our use-case for Tox

Here’s an excerpted version of Hypothesis’s tox.ini file.
It defines a few different testenvs named dev (for running the dev server),
tests (for running the tests), docs (for building the docs), etc. One
testenv for each development task. And tells Tox what dependencies to install
and what commands to run in each testenv. The rest of this tutorial will cover
everything you need to understand this tox.ini file in detail.

Installing Tox

Tox is like both GNU Make and your text editor in that you should install Tox
system-wide and use that one single copy of Tox for each project that you work
on. You don’t install Tox in your project’s virtualenv, or list it as a
dependency in your project’s requirements files. Tox is the thing that
creates your project virtualenvs and installs their dependencies for you.

On Ubuntu, just install Tox once system-wide and be done with it:

$ sudo pip install tox

Getting started: a minimal tox.ini file

Here’s the smallest possible tox.ini file that will run without crashing:

[tox]skipsdist=true

If you save this as tox.ini in an empty directory and then run the command
tox in that directory you should see something like this:

Tox created a virtualenv (at .tox/python) using the default version of Python
on your system (Python 2.7 on my Ubuntu system). Since the tox.ini file
didn’t list any dependencies or commands Tox didn’t install any dependencies or
run any commands in the virtualenv. It just did nothing.

The skipsdist = true
tells Tox not to try and build an sdist (source distribution) of your project.
An sdist is a way of packaging up a Python project so that it can be installed
on a system, published to https://pypi.org/, etc. Since our project at this
point is an empty directory there’s nothing to package up. At Hypothesis we
don’t build and publish installable Python packages anyway – we build web apps
that we deploy to production environments – so we always use
skipsdist = true.

Commands

Lets add a command to our tox.ini file using the commands setting.
We’ll make it print out the version of Python being used:

[tox]skipsdist=true[testenv]commands=python --version

Now if you run tox you’ll see the output of our command that it ran in the virtualenv:

We’ve now split the tox.ini file into two sections [tox] and [testenv].
The [tox] section contains global settings
that affect the global behavior of Tox itself, such as whether or not to build
an sdist, the minimum version of Tox that this tox.ini file requires, etc.

The [testenv] section defines the testenv(s) that we want Tox to create for
us: what versions of Python to use to create the testenvs, what dependencies to
install into them, what environment variables, what commands to run in the
testenvs, etc.
We’re gonna cover the most important settings that you can put in the
[testenv] section of your tox.ini file, starting with commands. For full
documentation of all the available settings run tox --help-ini or see
the docs.

External commands and whitelist_externals

One of the things that Tox does to isolate testenvs from the system is to
prepend the virtuenv’s bin directory onto the PATH envvar in the subshell
that the testenv’s commands are run in. You can see this by having Tox run a
command that prints out the value of PATH:

The PATH envvar in the testenv’s subshell is my usual PATH envvar but with
the .tox/python/bin dir prepended to the front. This is so that executables
in the virtualenv’s bin are always chosen instead of any executables with the
same name elsewhere on the system PATH. This is why when we’ve been putting
python --version in our tox.ini file its been printing out the version of
Python installed in the virtualenv instead of the system version of Python:
it’s running the copy of python at .tox/python/bin/python. Similarly pip
in a tox.ini file (or any file run by a tox.ini file) will be the
virtualenv’s copy of pip, and the same goes for any executables that your
package installs.

Notice that Tox printed out a test command found but not installed in testenv
warning. This is because the tox.ini file is running the executable sh
which is a system executable – there’s no sh in the virtualenv. By default
Tox allows executables from the system PATH to be run if they aren’t shadowed
by a virtualenv executable, but it prints this warning. We can silence it with
the whitelist_externals
setting, which is a list of allowed system executables:

Running multiple commands in sequence

You can easily have Tox run a sequence of commands in order by using multiple
commands in the commands setting, one command per line:

[tox]
skipsdist = true
[testenv]
whitelist_externals = echo
commands =
echo This is the first command
echo This is the second command
echo This is the third command

Many settings in Tox can take a list of lines like this, including
whitelist_externals, deps, passenv and setenv, etc.

Command exit codes and Tox’s exit code

If one of the commands exits with a non-zero exit code Tox will stop there
and not continue on to the next commands. For example the UNIX command false
exits with code 1. This tox.ini file will stop at false and not continue
on to the final command:

When a command from any one of the testenvs run exits with nonzero then Tox
itself will also exit with nonzero, whereas if all commands exited with zero
Tox will exit with zero.

You can tell Tox to ignore the exit code of a command – continuing on to the
next command even if that command exits with zero, and not failing the tox
command itself if that command fails – by prepending the command with a -.
This tox.ini file will run all three commands and exit with zero:

Passing command line arguments to commands using posargs

You can use Tox’s posargs substitution to allow the user to pass command line
arguments through to the command(s) that Tox is running. This is particularly
useful when using Tox to run a test runner, such as pytest, that has many
useful command line arguments. The user can run tox -- -x to run pytest
with pytest’s -x option, for example.

Here’s a tox.ini file that installs and runs pytest and passes any command
line args through:

(Here we’re using the deps setting to tell Tox to install pytest into the
virtualenv before running the command(s). More on deps later.)

If you just run tox this will have pytest run all the tests in your tests/
dir (it’ll run pytest tests/). If you want to pass any command line arguments
to pytest you put them after a -- in the tox command. If no -- is given
then {posargs} evaluates to the empty string. For example to have pytest
print out all its command line arguments and options:

$ tox -- --help

You can also give a default value for posargs following a : within the {…}.
When no -- is used on the command line the default value will be substituted in place of
the {posargs:DEFAULT_VALUE}. For example this will run pytest tests/ by
default but allow the user to narrow down what tests to run with a command like
tox -- tests/foo/bar:

commands=pytest {posargs:tests/}

{posargs} can be used anywhere in in tox.ini and can even be used multiple
times to pass the same positional arguments to multiple commands.

envlist

So far Tox has just been creating a single virtualenv for us named python,
using the system’s default version of Python. By adding the envlist setting
we can get it to create virtualenvs using different versions of Python and to
create and run multiple testenvs at once:

This requires Python 3.6 to be installed on the system where tox is run. You
need Python 3.6 to create a Python 3.6 virtualenv. The envlist setting is a
comma-separated list of testenvs that will be run when you run tox. If you
want to run your commands in multiple versions of Python at once you can just
give an envlist with multiple Pythons:

The envlist setting is the default testenv(s) to run when tox is run with
no -e argument. envlist can be overridden using the -e command line
argument. tox -e py27,py36 will run the py27 and py36 testenvs
regardless of what the envlist setting in tox.ini says.

You can also override envlist by setting the TOXENV environment variable,
and you can set the envvsr TOX_SKIP_ENV to a regular expression and any testenvs
that match the regex will be skipped.

Dependencies

You can have Tox install the dependencies that your commands need into the
virtualenvs before running the commands, using the deps setting
in tox.ini. For example this tox.ini file will install pytest and all
the requirements listed in your app’s requirements.txt file and then run
pytest:

If the deps setting in your tox.ini file changes Tox will automatically
recreate the virtualenv and reinstall the dependencies the next time you run
it.

By default, though, Tox doesn’t detect changes to dependencies listed in files when things like
-r requirements.txt are used. You would have to tell Tox to recreate the virtualenv (with the tox -r option)
whenever requirements.txt changes. Fortunately the venv_update extension, part of the
tox-pip-extensions package, automates this.

Environment variables

Another thing Tox does to help isolate testenvs from the system is that it
removes most of your shell’s environment variables from the subshell that it
runs the testenv commands in. You can see this by having a tox.ini file print
out all the environment variables available in the testenv using the standard
UNIX env command:

A handful of environment variables are allowed to pass through by default,
including PATH (with the virtuenv’s bin dir prepended). You can also see
that
Tox injects a few environment variables of its own
that you can make use of, such as TOX_ENV_NAME (the name of the testenv being run).

If you want to pass more envvars from your shell through, for example to
configure your application, you can do this using the passenv setting
which is a list of additional envvars that can be set in the environment where
tox is being run and will be passed in to the testenv where the commands are
run:

[testenv]passenv=DATABASE_URLELASTICSEARCH_URLPYTEST_ADDOPTS…

You can also use setenv
to set the values of additional envvars in the tox.ini file, rather than
reading them from the outside environment:

Substitutions

As well as being available to the commands that get run, environment variable
values can also be used in the tox.ini file itself by using Tox substitution.

Anywhere in tox.ini (for example in a dependency, or in a command) you can
use {env:ENVVAR_NAME} to substitute the value of an environment variable.

For example this tox.ini file allows you to set the command to be run by setting an environment variable named COMMAND.
Test it out with a command like env COMMAND="echo hi" tox:

[tox]skipsdist=true[testenv]passenv=COMMANDcommands={env:COMMAND}

If no COMMAND envvar is present Tox will exit with an error. You can provide
a default value for this case using a second ::

commands={env:COMMAND:pytest}

Substitutions can be nested, which can allow one envvar to be used as a fallback for another.
This will read the command from the COMMAND envvar or, failing that, from FALLBACK_COMMAND,
or finally default to pytest if neither envvar is set:

This tox.ini file uses Sphinx to build a project’s documentation (located in
the docs dir) and serve it locally for previewing. It uses {envtmpdir} as
the directory for Sphinx’s output files (the built documentation HTML files) to
avoid messing up the user’s working directory, and so that those files are
automatically cleaned up each run:

Factors and conditionals

Tox’s factors
and the conditional settings they enable are one of the most useful and flexible features
of Tox. They’re the basis of our tox.ini file at Hypothesis.
But to understand them you have to grock a few Tox concepts first:

Testenvs

A Tox testenv is a bunch of settings that define how to create and run
a virtuenv: what version of Python to use, what dependencies to install, what
environment variables to set and to pass through, what commands to run in the
venv once it's ready, etc.

Whenever you run tox you're running one or more testenvs
(the testenvs listed in the envlist setting, -e command line
argument, or TOXENV environment variable).

Testenv names are lists of factors

A testenv name is a minus-separated list of factors.
A testenv named py27 contains a single factor, py27.
A more complex testenv name like py27-tests contains two factors
py27 and tests.

Factors

A factor is a collection of testenv settings. A testenv like
py27-tests pulls in all of the settings from the py27
factor and the tests factor, and those settings combined become the
testenv's settings.

py27, py36 etc are builtin factors.
Tox comes with builtin factors for all the versions of Python, and you can just
use these in your tox.ini files.
These are factors that contain a single setting: the version of Python to use to
create the virtualenv with
(the basepython setting).

You can also define custom factors in your tox.ini file. test in the
testenv name py27-test is a custom factor.

Conditionals

Whenever you give a list of things in your tox.ini file (for
example the lists of environment variables to pass through, dependencies to
install, and commands to run) items in the list can be made conditional on a factor
by prefixing them with factorname: .
For example in this tox.ini snippet:

[testenv]
deps =
tests: pytest

the tests: pytest implicitly creates a custom factor named tests
and tells Tox to install the pytest dependency only when the tests factor
is invoked. tox -e py27-tests will install pytest but tox -e py27-docs won't.

Below is a simplified version of the Hypothesis tox.ini file
that uses factors and conditionals to define separate testenvs for running the
dev server, running the linter, running the tests, and building the docs. This
enables the following tox commands:

tox runs the tests in Python 2.7 (py27-tests is the default envlist)

You can run the tests in any version of Python (as long as you have that version
of Python installed on your system) by using different testenv names:
tox -e py27-tests runs the tests in Python 2.7, whereas tox -e py36-tests
or tox -e py37-tests run the tests in Python 3.6 or 3.7.

tox -e py27-dev runs the dev server in Python 2.7.
As with the tests you can run the dev server in any version of Python using
commands like tox -e py36-dev.

As you can see the tox.ini file is designed to enable pyXY-COMMAND
testenvs, where pyXY is any version of Python and COMMAND is any of the
provided commands (tests, dev, lint, etc). It can be extended by adding
as many commands as you like, automating all of your development tasks.

Tox reads the commands list from top to bottom and runs only the commands
that match the testenv name. python --version is an unconditional command (no
factor: prefix), so that always gets run. The two dev: commands match the dev
factor in py36-dev so they get run. The other commands
conditions don’t match so they’re ignored. The matching commands are run in the
order that they appear in the file.

This same matching procedure is applied to all lists in the tox.ini file:
passenv, deps, etc. When you run tox -e py36-dev Tox first collects all
the settings in the builtin py36 factor (the Python version: 3.6), then it
parses the tox.ini file and collects all settings that are either
unconditional or whose condition matches one of the factors py36 or dev.
These collected settings form the testenv definition, and Tox runs it.

More complex conditions are also possible. The most useful of these is to
reduce duplication by making a single line match multiple factors by giving a
comma-separated list of factors in {…}’s. This will install
requirements.txt if either the tests or the dev factor is invoked:

Debugging your tox.ini file with --showconfig

Once you’ve started adding conditionals to your tox.ini file it can be useful
to have a good way to debug it. Given a tox.ini file like the one above, and
a command like tox -e py27-dev, adding the --showconfig option will get Tox
to print out the entire collected py27-dev testenv definition instead of
running the testenv’s commands:

Generative envlists

So far we’ve mostly ran one or two testenvs at a time with commands like tox
-e py36-tests or default envlist settings like envlist = py27-tests,py36-tests.
It isn’t much use to us at Hypothesis but for completeness sake Tox also has a
Generative envlist
feature that allows you to run many testenvs at once without having to spell
out every single testenv name. This is useful for projects that need to support
a lot of combinations of different versions of Python, different versions of
dependencies, and different platforms at once. A single tox command can run
your tests in with all the different combinations. Here’s an example from the docs:

envlist={py27,py36}-django{15,16}, docs, flake

With this envlist setting a single tox command will generate the
“cross product” of the different versions of Python and Django and run
each possible combination. It will run all of these testenvs:

py27-django15

py27-django16

py36-django15

py36-django16

docs

flake

Defining testenvs using [testenv:NAME] sections

So far we’ve had a single [testenv] section in our tox.ini and we’ve used
conditionals to define multiple testenvs. There’s also an entirely different
way to define your testenvs in your tox.ini: using a separate [testenv:NAME]
section for each testenv. Here’s an example of a tox.ini file that supports
tox or tox -e py27 to run the tests in Python 2.7, and tox -e py27-dev
and tox -e py36-dev to run the dev server in Python 2.7 or 3.6:

[tox]envlist=py27-testsskipsdist=true[testenv]skip_install=true[testenv:py27-tests]description=Run the tests in Python 2.7deps=coveragemock…-rrequirements.txtpassenv=TEST_DATABASE_URLELASTICSEARCH_URLPYTEST_ADDOPTScommands=coveragerun--parallel--sourceh,tests/h-mpytest-Werror{posargs:tests/h/}[dev]deps=-rrequirements-dev.inpassenv=ALLOWED_ORIGINSAUTHORITYBOUNCER_URL…whitelist_externals=shcommands=shbin/hypothesis--devinit{posargs:shbin/hypothesisdevserver}[testenv:py27-dev]description=Run the dev server in Python 2.7deps={[dev]deps}passenv={[dev]passenv}whitelist_externals={[dev]whitelist_externals}commands={[dev]commands}[testenv:py36-dev]description=Run the dev server in Python 3.6deps={[dev]deps}passenv={[dev]passenv}whitelist_externals={[dev]whitelist_externals}commands={[dev]commands}

Any settings in the [testenv] section apply to all testenvs, unless
overridden. You then define each testenv in its own section, for example the
[testenv:py27-tests] section contains the settings for the py27-tests env
(tox -e py27-tests).

The [dev] section doesn’t define a testenv and isn’t used by Tox. It’s just a bag of settings
to be used in substitution by the [testenv:py27-dev] and [testenv:py36-dev] sections below.
For example when [testenv:py27-dev] does deps = {[dev]deps} that means pull
in all the deps from the [dev] section. You can also pull in some deps and then add more:

[testenv:py27-dev]deps={[dev]deps}ipythonipdb

These are called section substitutions.
Using substitutions like this is a way to avoid duplication. It’s almost like
inheritance, where [testenv:py27-dev] and [testenv:py36-dev] are child
sections and [dev] is the base section they inherit from. But there’s no way
to automatically inherit all the settings from a base section at once: you have
to inherit each section individually with a substitution.

One advantage of writing your tox.ini file this way is that you can add a
description to each testenv, and if you run tox -av it’ll list all the
available testenvs and their descriptions:

$ tox -av
using tox.ini: /home/seanh/Projects/h/tox.ini
using tox-3.5.2 from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc
default environments:
py27-tests -> Run the tests in Python 2.7
additional environments:
py27-dev -> Run the dev server in Python 2.7
py36-dev -> Run the dev server in Python 3.6

Overall, though, we found this way of writing tox.ini files produced much
larger files that contained a lot more boilerplate and duplication, and were
more difficult to read and to work with. The file above only supports running
the dev server in Pythons 2.7 and 3.6, for example. To run it in Python 3.7
you’d have to add a new [testenv:py37-dev] section.

The factors and conditionals approach produces a much smaller tox.ini file
while also enabling any command to be run with any version of Python.

Grab bag

Some other cool stuff that Tox can do:

Interactive shell substitution.
You can use substitutions like {tty:ON_VALUE:OFF_VALUE} to have ON_VALUE be used when Tox
is run in an interactive shell (e.g. on a developer’s laptop) and OFF_VALUE be used when its
run non-interactively (e.g. on CI). You could use this, for example, to pass the --pdb argument
to pytest to make it drop in to a debugger prompt when a test fails, but only when Tox is
being run interactively.

Jenkins support. You can add a [tox:jenkins] section containing global settings
overrides that should be used only when Tox is running in Jenkins.
See Jenkins override.

Force a particular version of a dependency just once with --force-dep.