Fork

When running instrumentation tests, there is a significant time overhead for developers, especially on larger test suites. Existing solutions were not satisfactory for quick feedback before pushing code to VCS and for CI purposes.

We are big fans of Spoon and were using it for our plans, so we used it as our starting point. However, Spoon had similar issues to the Gradle and Maven test execution plugins, in the sense that it executes all tests on all of the connected devices (and emulators). We decided to tweak that naive scheduling to achieve much faster test execution.

How it works

We introduced the notion of pools of devices. These are now responsible for running a test suite instead of each device running the suite separately. That has two side effects:

infinite scaling: your tests can speed up by as many devices and emulators as you can dedicate to your CI box.

because test suites now get scheduled to run on a pool, not all tests will run on all devices. For that reason, we also introduced a way to create a pool per device, which offers full coverage (a.k.a. Spoon-mode) but typically takes longer, so we run it on a nightly basis.
Fork works out-of-the-box, without any code changes.

Running Fork

There are two ways to run Fork with your builds.

Gradle plugin (recommended)

First, you need to add a build-script dependency. You can have access to snapshot builds, but stable versions are recommended.

You're now done. If you had any instrumentation test tasks before, the plugin has added Fork tasks. You can verify by running:

gradlew tasks | grep fork

You can use Fork's DSL to configure its execution parameters. For a full list of the properties, have a look at: Configuring pools and runtime and related Examples. It should be as easy as adding a block to your build.gradle:

Configuring pools and runtime

One of the most useful characteristics of the library is the way it creates the device pools. There are different options, to automatically create pools by API level, shortest width dimension and whether devices are self-described as tablets. On top of that, users can also manually create pools based on serial numbers, for maximum flexibility.

With either way of executing Fork (Gradle / Standalone) you can specify how the pools are created by setting a combination of the properties below. You can also find examples:

Property Name

Property Type

Default value

baseOutputDir

File

"fork"

ignoreFailures

boolean

false

isCoverageEnabled

boolean

false

testClassRegex

String

"^((?!Abstract).)*Test$"

testPackage

String

(Your instrumentation APK package)

title

String

-

subtitle

String

-

testOutputTimeout

int

60000

testSize

String

-

excludedSerials

Collection<String>

-

fallbackToScreenshots

boolean

false

totalAllowedRetryQuota

int

0

retryPerTestCaseQuota

int

1

autoGrantPermissions

boolean

true

poolingStrategy

PoolingStrategy

-

excludedAnnotation

String

(tests with this annotation are excluded)

Poolingstrategy is an object that describes how the device pools are created. You can choose only one strategy from below:

map pool names to collection of device serials to be assigned to that pool

ComputedPooling's properties are:

Property Name

Property Type

Description

characteristic

Characteristic

Possible values: "sw"\

"api" (shortest width or API level)

groups

Map<String, Integer>

map the name of a pool to their lowest dimension for a characteristic

Runtime Permissions

By default Fork auto-grants all runtime permissions on Android Marshmallow +. It is possible anyway to selectively revoke one or more permissions per single test case.
To do so, you have to add an annotation called RevokePermission. Here is an example:

Remember to add the fork client-side library to your project to have access to the annotation.
To do so, in your app's dependencies add:

androidTestImplementation "com.shazam.fork:fork-client:3.6.0"

After every test case, all the runtime permissions will be automatically re-granted even if the test fails.
This feature will impact only Marshmallow and subsequent devices.

Arbitrary metadata

Using Fork you can set metadata on tests and get them back in its JUnit xml reports. The metadata are added as additional property tags on the suite level of the report, as each test produces its own report.

Limitations

The scheduling still works on a single build box with ADB, so there still is a limit by how many devices & emulators can be simultaneously connected to ADB. Eventually, Fork could be tweaked to talk over HTTP with other build agents, that would then be connected to devices over ADB. That model would tie in nicely with multi-agent CI systems, like Jenkins.

Flakiness Reporter

One common problem with UI tests is the test flakiness from either the environment they run on or badly written tests. To help track down tests that are misbehaving, we introduced the Flakiness reporter.

The reports produced by the Flakiness Reporter eventually make it trivial to find flaky tests and link to them and their diagnostics. Currently Jenkins is supported and it should be really easy to extend it to other types of CI servers.

How it works

The Flakiness Reporter collects Fork output files, matches test runs over previous builds and sorts them according to their flakiness. Links are also created to each test of each test run, for easy navigation to diagnostics.

Sample output

The output after a successful run of the Flakiness Reporter looks like the following:

Running the Flakiness Reporter (Jenkins)

The Gradle plugin that allows the Reporter to run can be applied to a standalone project, since it doesn't directly depend on your Android project. For convenience, however, that is a good compromise.

Currently, the Reporter supports Jenkins but plugins can be written to be used with other CI servers.

Chimprunner

At the time of writing, not much is available around automated performance testing. Chimprunner is a very simple test runner that allows recording of somewhat accurate timings on test execution, from process creation to test finish. It all works on the Android instrumentation tests system that developers are familiar with.

Current reports

Currently, Chimprunner produces a timings.csv file in the output folder with all the timings of tests that were executed as part of the performance tests and the average time they took after running a number of iterations. That CSV file can be then used for plotting by other tools. Using the Jenkins Plot plugin we can now produce historic diagrams of our startup time like the following diagram:

Future work

We would like to add ways of automatically launching Android performance tools & reports developers know and use already, with no or minimal code changes. We will investigate around how to provide systrace, CPU, GPU & memory usage reports. The library will probably provide some annotations that will enable various performance tools. An example could be:

The system could then provide a systrace & GPU profiling reports for the duration of the trackListScroll() test.

License

Copyright 2016 Shazam Entertainment Limited.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.