autopkgtest (1)

NAME

SYNOPSIS

DESCRIPTION

autopkgtest runs tests on binary Debian or Click packages, as installed on a
system (called "testbed"). The tests are supplied in the source package.
autopkgtest runs each test supplied by a particular package and reports
the results. It drives the specified virtualisation regime as
appropriate, parses the test description metadata, and arranges
for data to be copied to and from the testbed as required.
See /usr/share/doc/autopkgtest/README.running-tests.rst.gz for an
introduction about how to use autopkgtest.

TESTING A DEBIAN PACKAGE

Positional (non-option) arguments specify exactly one source package
(containing the test code) and optionally some binary packages to test.
testsrc can be one of:

.dsc file

Run tests from Debian .dsc source package. By default the package will
also be built and the resulting binaries will be used to satisfy test
dependencies; to disable that, specify the
-B/--no-built-binaries
option.

source package directory

Run tests from a Debian source tree directory. If that is an unbuilt tree,
this is very similar to specifying a .dsc. If that is a built tree, all
test dependencies get satisfied by archive packages, unless you explicitly
specify locally built .debs as well.
Attention:
If you just specify a bare directory name which is a legal Debian source
package name, it will be interpreted as the latter (see below). In this case,
prefix the directory name with
./.

current directory

If no source package is specified on the command line and the current directory
is a Debian source package, this will be tested.

source package name

Downloads the given source package name with apt-get source in the
testbed and run its tests. This is similar to specifying a .dsc
but avoids copying the source from the host to the testbed. Possibly built
binaries (if the test specifies
needs-build) will
not
be used to satisfy dependencies, as usually in this mode you want to test
binaries from a real archive.

git URL or URL#branch

Git-clones the given URL (which must contain an unbuilt Debian source tree) and
runs the tests from that.
If
branch
is given, this branch will be checked out instead of the default (usually
"master"). This can also be a more general refspec such as #refs/pull/123/head"
for a GitHub pull request.
This is very similar to cloning manually and specifying the checkout directory
as test; i. e. this is commonly used with
--no-built-binaries.
The
git
package will be installed if necessary.

.changes file

Run tests from the .dsc source package in the given .changes file. If
the .changes contains .deb packages, they will be used for the test.
Acts as if you had specified the .debs and .dsc from the .changes file as
explicit arguments. Note that if the .changes contains
only
debs, the corresponding .dsc still needs to be specified alongside, or the
current directory must be the source package.

All other positional arguments must be .deb binary packages. They will be used
for both build and test dependencies of the source package. If any binary
package is given, then
--no-built-binaries
is implied.

TESTING A CLICK PACKAGE

This needs exactly one .click package. If its manifest specifies an
x-source
URL, the corresponding source package will be downloaded and tests run from
there; then a source directory does not need to be given. Otherwise specifying
a source directory as positional argument is mandatory.
For testing a local .click package, give its path to
autopkgtest.
For testing a preinstalled click, use
--installed-click com.example.myclick
(this cannot be given as positional argument as this cannot be disambiguated
from Debian source package names).
Examples with explicit click source directory:

TEST OPTIONS

-B | --no-built-binaries

Binaries from unbuilt source packages (see above)
will not be built or ignored, and dependencies are satisfied with packages from
the archive. Note that the source package still gets built if a test requires
build-needed.

--override-control=PATH

Read the test metadata from
PATH
instead of
debian/tests/control
(for Debian sources)
or the Click manifest.

--test-name=TEST

Run only the given test name (from test control file). This replaces
--testname which is deprecated.

LOGGING OPTIONS

If you don't specify any option,
autopkgtest
only writes its output/results to stderr.

-o dir | --output-dir=dir

Specifies that test artifacts (stderr and stdout from the tests, the log file,
built binary packages etc.) should be placed in the given directory.
dir must not exist yet or be empty, otherwise
autopkgtest
will refuse to use it.

-l logfile | --log-file=logfile

Specifies that the trace log should be written to logfile
instead of to output-dir.

--summary=summary

Specifies that a summary of the outcome should be written to
summary. The events in the summary are written to the log
in any case.

-q | --quiet

Do not send a copy of autopkgtest's trace logstream to stderr. This
option does not affect the copy sent to logfile or
output-dir. Note that without the trace
logstream it can be very hard to diagnose problems.

TEST BED SETUP OPTIONS

--setup-commands=commands

Run
commands
after opening the testbed. This can be used e. g. to enable additional apt
sources, run
apt-get update
or similar.
If
commands
is an existing file name, the commands are read from that; otherwise it
is a string with the actual commands that gets run as-is. File names
without directory will be searched in both the current directory and in
/usr/share/autopkgtest/setup-commands/
so you do not need to give the full path for setup scripts shipped with
autopkgtest.
Normally, if the setup commands fail, autopkgtest will consider this a
transient testbed error (exit code 16). However, if the setup commands exit
with code 100, autopkgtest will consider this an "erroneous package" (exit code
12) instead, so this can be used to e. g. detect upgrade errors to a new
version. Note that apt exits with exit code 100 in these cases.
This option can be specified multiple times.
If
--user
is given or the test bed provides a
suggested-normal-user
capability, the
$AUTOPKGTEST_NORMAL_USER
environment variable will be set to that user.
If the setup commands affect anything in boot directories (like /boot or
/lib/systemd/system) and the testbed supports rebooting, the testbed will be
rebooted after the setup commands. This can be suppressed by creating a file
/run/autopkgtest_no_reboot.stamp.

--setup-commands-boot=commands

Run
commands
after the
--setup-commands,
and after every reboot. For example, these commands could be used to
add files in a tmpfs.
These commands never cause the testbed to be rebooted (because that could
lead to an infinite loop). Otherwise, they are just like the
--setup-commands.
This option can be specified multiple times.

--apt-upgrade | -U

Run
apt-get update
and
apt-get dist-upgrade -y
in the testbed before running the tests.

--apt-pocket=pocket[=pkgname,src:srcname,...]

Add apt sources for release-pocket. This finds the first
deb
line in
/etc/apt/sources.list
which does not already specify a pocket and adds a deb and deb-src line with
that pocket to
/etc/apt/sources.list.d/pocket.list.
This also calls
apt-get update
for the new pocket (but not for anything else).
If a package list is given after =, set up apt pinning to use only those
packages from
pocket.
An entry "src:srcname" expands to all binary packages built by that source.
This can be used for minimizing dependencies taken from
pocket
so that package updates in that pocket can be tested independently from each
other for better isolation.
Attention:
This does not currently resolve some situations where dependencies of
the given packages can only be resolved in the given pocket. In that case the
apt pinning will be removed and package installation will be retried with the
entirety of
pocket.

--copy=HOSTPATH:TESTBEDPATH

Copy file or directory from host into testbed after opening. This happens
before
--setup-commands
thus you can use these files in the setup commands.

--env=VAR=value

Set arbitrary environment variable in the build and test. Can be specified
multiple times.

USER/PRIVILEGE HANDLING OPTIONS

-u user | --user=user

Run builds and tests as user on the testbed. This needs root on
the testbed; if root on the testbed is not available then builds and
tests run as whatever user is provided.

--gain-root=gain-root

Prefixes
debian/rules binary
with
gain-root.
The default is not to use anything, except that if
--user is supplied or root on the testbed is not available the
default is fakeroot.

DEBUGGING OPTIONS

--debug|-d

Include additional debugging information in the trace log. Each
additional -d increases the debugging level; the current maximum
is -ddd. If you like to see what's going on, -d or
-dd is recommended.

--shell-fail

|-s

Run an interactive shell in the testbed after a failed build, test, or
dependency installation.

--shell

Run an interactive shell in the testbed after every test.

TIMEOUT OPTIONS

--timeout-which=seconds

Use a different timeout for operations on or with the testbed. There
are five timeouts affected by five values of which:
short:
supposedly
short operations like setting up the testbed's apt and checking the
state (default: 100s);
install:
installation of packages including dependencies
(default: 3,000s);
test:
test runs (default: 10,000s);
copy:
copy files/directories between host and testbed
(default: 300s); and
build:
builds (default:
100,000s). The value must be specified as an integer number of seconds.

--timeout-factor=double

Multiply all of the default timeouts by the specified factor (see
--timeout-which above). Only the defaults are affected;
explicit timeout settings are used exactly as specified.

LOCALE OPTIONS

--set-lang=langval

When running commands on the testbed, sets the LANG environment
variable to langval. The default in autopkgtest is to set it
to C.UTF-8.

OTHER OPTIONS

--no-auto-control

Disable automatic test generation with autodep8, even if it is installed. In
that case, packages without tests will exit with code 8 ("No tests in this
package") just like without autodep8.

--build-parallel=N

Set
parallel=N
DEB_BUILD_OPTION for building packages. By default this is the number of
available processors. This is mostly useful in containers where you can
restrict the available RAM, but not restrict the number of CPUs.

-h|--help

Show command line help and exit.

VIRTUALIZATION SERVER

--virt-server virt-server-arg...

Specifies the virtualisation regime server, as a command and arguments
to invoke.
virt-server
must be an existing autopkgtest virtualization server such as
schroot
or
qemu.
All the remaining arguments and options after
--
are passed to the virtualisation server program. See the manpages of the
individual servers for how to use them.

OUTPUT FORMAT

During a normal test run, one line is printed for each test. This
consists of a short string identifying the test, some horizontal
whitespace, and either
PASS
or
FAIL reason
or
SKIP reason
where the pass/fail indication is separated by any reason by some
horizontal whitespace.
The string to identify the test consists of a short alphanumeric
string invented by autopkgtest to distinguish different command-line
arguments, the argid, followed by a hyphen and the test name.
Sometimes a
SKIP
will be reported when the name of the test is not known or not
applicable: for example, when there are no tests in the package, or a
there is a test stanza which contains features not understood by this
version of
autopkgtest.
In this case
*
will appear where the name of the test should be.
If autopkgtest detects that erroneous package(s) are involved, it
will print the two lines
blame: blamed-thing...
and
badpkg: message.
Here each whitespace-separated blamed-thing is one of
arg:argument
(representing a pathname found in a command line argument),
dsc:package
(a source package name),
deb:package
(a binary package name)
or possibly other strings to be determined. This indicates which
arguments and/or packages might have contributed to the problem; the
ones which were processed most recently and which are therefore most
likely to be the cause of a problem are listed last.

CONFIGURATION FILES

If you use lots of options or nontrivial virt server arguments, you can put any
part of the command line into a text file, with one line per option. E. g. you
can create a file
sid.cfg
with contents like

-s
--output-dir=/tmp/testout
--apt-upgrade
--
schroot
sid

and then run

autopkgtest foo_1_amd64.changes @sid.cfg

The contents of the configuration file will be expanded in-place as if you
would have given its contents on the command line. Please ensure that you
don't place spaces
between short options and their values, they would become a part of the
argument value.

EXIT STATUS

0all tests passed

2at least one test skipped

4at least one test failed

6at least one test failed and at least one test skipped

8no tests in this package

12erroneous package

16testbed failure

20other unexpected failures including bad usage

SEE ALSO

AUTHORS AND COPYRIGHT

This manpage is part of autopkgtest, a tool for testing Debian binary
packages. autopkgtest is Copyright (C) 2006-2014 Canonical Ltd.
See /usr/share/doc/autopkgtest/CREDITS for the list of
contributors and full copying conditions.