Write a Tradefed Test Runner

Background

This is not a prerequisite to writing a new test runner; test runners can be
written in isolation.

Bare minimum: Implementing the interface

The bare minimum to qualify as a Tradefed test runner is to implement the
IRemoteTest interface
and more specifically the run(ITestInvocationListener listener) method.

This method is the one invoked by the harness when using the test runner,
similar to a Java Runnable.

Every part of that method is considered part of the test runner execution.

Reporting results from the test runner

The run method in the base interface give access to a listener object of
type ITestInvocationListener. This object is the key to reporting structured
results from the test runner to the harness.

By reporting structured results, a test runner has the following properties:

Report a proper list of all the tests that ran, how long they took and if
they individually passed, failed or some other states.

Report metrics associated with the tests if applicable, for example
installation-time metrics.

Fit in most of the infrastructure tooling, for example display results and
metrics, etc.

Usually easier to debug since there is a more granular trace of the
execution.

That said, reporting structured results is optional; a test runner might
simply want to assess the state of the entire run as PASSED or FAILED without
any details of the actual execution.

NOTE: It is more difficult to implement a runner that follows the sequence of
events, but we do recommend doing so given the benefits listed above.

The following events can be called on the listener to notify the harness of the
current progress of executions:

testRunStarted: Notify the beginning of a group of test cases that are
related together.

testStarted: Notify the beginning of a test case starting.

testFailed/testIgnored: Notify the change of state of the test case
in progress. A test case without any change of state is considered
passed.

testEnded: Notify the end of the test case.

testRunFailed: Notify that the overall status of the group of test cases
execution is a failure. A test run can be a pass or a failindependently of the test cases results depending on what the
execution was expecting. For example, a binary running several test cases
could report all pass test cases but with an error exit code (for any
reasons: leaked files, etc.).

testRunEnded: Notify the end of the group of test cases.

Maintaining and ensuring the proper order of the callbacks is the
responsibility of the test runner implementer, for example ensuring that
testRunEnded is called in case of exception using a finally clause.

Testing with a device

The minimum interface above allows to run very simple tests that are isolated
and do not require any particular resources, for example Java unit tests.

Test writers who want to go to the next step of device testing will need the
following interfaces:

IDeviceTest
allows to receive the ITestDevice object that represents the device under
test and provides the API to interact with it.

IBuildReceiver
allows the test to get the IBuildInfo object created at the
build provider step
containing all the information and artifacts related to the test setup.

Test runners are usually interested in these interfaces in order to get
artifacts related to the execution, for example extra files, and get the
device under test that will be targeted during the execution.

Testing with multiple devices

Tradefed supports running tests on multiple devices at the same time. This is
useful when testing components that require an external interaction, like a
phone and a watch pairing.

In order to write a test runner that can use multiple devices, you will need
to implement the
IMultiDeviceTest,
which will allow to receive a map of ITestDevice to IBuildInfo that contains
the full list of device representations and their associated build information.

The setter from the interface will always be called before the run method, so
it's safe to assume that the structure will be available when run is called.

Tests aware of their setups

NOTE: This is not a very common use case. It is documented for completeness,
but you will not usually need it.

Some test runner implementations might need information about the overall setup
in order to work properly, for example some metadata about the invocation, or
which target_preparer ran before, etc.

In order to achieve this, a test runner can access the IConfiguration object
it is part of and that it's executed in. See the
configuration object
description for more details.

For the test runner implementation, you will need to implement the
IConfigurationReceiver
to receive the IConfiguration object.

Flexible test runner

Test runners can provide a flexible way of running their tests if they have a
granular control over them, for example a JUnit tests runner can individually
run each unit test.

This allows the larger harness and infrastructure to leverage that fine control
and users to run partially the test runner via filtering.

Filtering support is described in the
ITestFilterReceiver interface,
which allows to receive sets of include and exclude filters for the tests
that should or should not run.

Our convention is that a test will be run IFF it matches one or more of the
include filters AND does not match any of the exclude filters. If no include
filters are given, all tests should be run as long as they do not match any of
the exclude filters.

NOTE: We encourage test runners to be written in a way that supports this
filtering as it provides a huge added value in the larger infrastructure. But
we understand that in some cases it's not possible to do so.

Content and code samples on this page are subject to the licenses described in the Content License. Java is a registered trademark of Oracle and/or its affiliates.