Get started with Firebase Test Lab from the gcloud Command Line

Firebase Test Lab provides cloud-based infrastructure for testing
Android apps, including full integration with the
gcloud command-line interface (CLI).
This document covers the installation and configuration required
to get started using Test Lab from the gcloud command line interface.

Configure your local Google Cloud SDK environment

Make sure your Cloud SDK installation is up-to-date and includes
the gcloud firebase command:

gcloud components update

Make sure your authentication credentials are current:

gcloud auth login

Set your current project in gcloud:

gcloud config set project PROJECT_ID

Choosing test configurations

In this example, you will run some tests on a simple note-taking Android app
named Notepad.

Download the binary APK file for the Notepad app
(app-debug-unaligned.apk)
and its corresponding instrumentation tests
(app-debug-test-unaligned.apk)
provided in the NotePad/app/build/outputs/apk/ directory of
notepad.zip.

Get the current list of Android devices available to test against, as
follows:

$ gcloud firebase test android models list

The first column of the command output, MODEL_ID, contains the identifier
that you can use later to run tests on a specific model. The OS_VERSION_ID
column lists the operating system versions supported by that device. If you
don't specify which MODEL_ID(s) to test against, the default noted under the
TAGS column is used.

Learn more about a specific Android MODEL_ID with the firebase test android
models describe command, as follows:

$ gcloud firebase test android models describe Nexus5

The example command shown above provides detailed information about the Nexus5
model, including the brand, manufacturer, and supported API levels, and whether
the model is physical or virtual.

Get the current list of Android OS versions available to test against:

$ gcloud firebase test android versions list

You can use an identifier from either of the first two columns of command output
(OS_VERSION_ID and VERSION), to later run tests against an Android OS
version. If you don't specify the Android OS versions to test against, the
default noted under the TAGS column is used.

Get the current list of locales available to test against:

$ gcloud firebase test android locales list

The first column of the command output, LOCALE, contains the identifier that
you can use later to run tests against a locale. If you don't specify the
locales to test against, English is used as the default locale. Command output
is not shown here because hundreds of locales are available.

Running tests

Now that you know the range of device models, OS versions, and locales available
to use when testing your app, you can use that information to specify test
devices using the gcloud firebase test android run command and the --device
flag. This command and flag are used whether you are using the Robo test to
automatically test your app, or whether you are running instrumentation tests
written specifically to test your app.

Note: The --device flag is now the preferred way to specify test devices and
may not be used in conjunction with --devices-ids, --os-version-ids,
--locales, or --orientations. To learn about these legacy flags, see the
list of deprecated device dimension flags at
gcloud firebase test android run.

Running the Robo test

Even if you don't have any instrumentation tests, you can still look
for bugs in your app. Use the Robo test to perform automated review of your
app's user interface. Robo test exercises the app by performing a static
analysis of the various paths through the app's user interface, and then
crawling through the app to find crashes and other potential issues.

The --type robo parameter is implicit if no --type value is specified.
You can see the complete set of command line options for running tests by
typing: gcloud help firebase test android run.
As an alternative to specifying all these arguments on the command line,
you can optionally specify your arguments in a YAML-formatted argument file.
Run gcloud topic arg-files to learn how to use this feature.

Running your instrumentation tests

Now use the gcloud command line tool to run the Notepad app's
Espresso
tests on your specified Android device configurations, using the
instrumentation test type to run the tests in app-debug-test-unaligned.apk
as follows:

The --type instrumentation parameter is implicit if a test APK has been
specified with --test. As an alternative to specifying all these arguments on
the command line, you can optionally specify your arguments in a YAML-formatted
argument file. Run gcloud topic arg-files to learn how to use this feature.

The gcloud CLI supports Android Test Orchestrator.
Orchestrator requires AndroidJUnitRunner v1.0 or higher. To enable it, use
gcloud firebase test android run with the --use-orchestrator
flag. To disable it use the --no-use-orchestrator flag.

Note: You can also control how Test Lab runs your instrumentation tests
using additional flags that are not shown above. For example, you can use the
--test-targets flag to test just one class used by your test APK, or to test
just one method from a class used by your test APK. To learn more, see
gcloud firebase test android run.

Code coverage reports for instrumentation tests

Test Lab supports code coverage reporting tools
EMMA and
JaCoCo. If you have either tool integrated
into the build for your app, you can get a code coverage report for Test Lab
tests by running the gcloud firebase test android run command with the
following arguments:

Open the Firebase console link that the gcloud tool printed above the
test result table in your terminal.

Click a test execution from the list at that link to open that execution's
details page.

Click Test results to go to the Google Cloud Storage bucket with that
execution's test results.

Open artifacts/coverage.ec to see your code coverage report.

Note: Pulling the sdcard directory for each test places all files from that
directory into the test's Google Cloud Storage bucket. If you're on a Flame or
Blaze payment plan and your app creates a lot of files in that directory,
pulling the directory could result in significant storage expenses.

Analyze your test results

After a few minutes, a basic summary of your test results is printed by the
gcloud tool:

Custom login and text input with Robo test

Robo test automatically completes sign-in screens that use a Google account
for authentication, unless you use the
--no-auto-google-login
parameter. It can also complete custom login screens using test account
credentials that you provide. You can also use this parameter to provide custom
input text for other text fields used by your app.

To complete text fields in your app, use the
--robo-directives
parameter and provide a comma-separated list of key-value pairs, where the
key is the Android resource name of the target UI element, and the value is
the text string.
EditText
fields are supported but not text fields in WebView UI elements.

Flaky Test Attempts flag:
a flag that specifies the number of times a test execution should be re-attempted if one or more of its test cases fail for any
reason. All additional attempts are executed in parallel.

Ignore Robo Directive Command:
a command that instructs the robo test to ignore any UI elements with resource names which equal or start with the user-defined values.

Note:Run Orchestrator locally before trying it in Test Lab. To avoid spending quota on or getting billed for malfunctioning tests, confirm that Orchestrator is working in your app by running a test on your own machine before uploading your APK. Also, testing with Orchestartor takes slightly longer than usual, and might impact your quota or billing.

Scripting gcloud commands with Test Lab

You can use shell scripts or batch files to automate mobile app testing commands
that you would otherwise run using the gcloud command line. The following
example bash script runs an instrumentation test with a two-minute timeout, and
reports if the test run completed successfully:

Script exit codes

Test Lab provides several exit codes that you can use to better understand the
results of tests that you run using scripts or batch files.

Scripting exit codes for Test Lab

Exit code

Notes

0

All test executions passed.

1

A general failure occurred. Possible causes include: a filename that does
not exist or an HTTP/network error.

2

Testing exited because unknown commands or arguments were provided.

10

One or more test cases (tested classes or class methods) within a test
execution did not pass.

15

Firebase Test Lab could not determine if the test matrix passed or
failed, because of an unexpected error.

18

The test environment for this test execution is not supported because of
incompatible test dimensions. This error might occur if the selected Android API
level is not supported by the selected device type.