Firebase Test Lab Game Loop Test

Automating game testing is challenging because of the wide range of UI
frameworks used for game development (some of which are engine-dependent), and
the difficulties of automating UI navigation in games. To support game app
testing, Test Lab now includes support for using a "demo mode" where the
game app runs while simulating the actions of a player. This mode can include
multiple loops (or scenarios), which can be logically organized using labels so
that you can run related loops together.

This document provides guidelines on how to implement game loops so that
you can easily use them to test your game during development. These
guidelines apply whether you are running tests on a single test device, on a
farm of test devices, or using Test Lab.

Get Started

To use the game loop test in Test Lab, your game must be modified to do the
following:

Launch the loop

Run the loop

(Optional) Close the game app

Launch the loop

When launching the game loop test, your game is triggered with a specific
intent, so you need to modify your manifest and add a new intent filter to your
activity, as shown in the following example code:

Note: Game loops require a separate, dedicated intent filter that is not used
for other purposes. Your activity can include other intent filters. To learn
more about intent filters, see
<intent-filter>.

Run the loop

Your activity, when launched, checks which intent launched it, as shown in the
following example code:

We recommend running this code in your onCreate method, but you can run it
later if you prefer (for example, after initially loading your game engine). You
can then implement your game loop (or multiple loops, as described in
Multiple game loops) however you like,
depending on your game engine. For example, game loops can be used to:

Run a level of your game the same way an end user would play it. You can
either script the input of the user, let the user idle, or replace the user
with an AI if it makes sense in your game (for example, if you already have
an AI implemented, like in a car racing game, and you can easily put an AI
driver in charge of the user's input).

Run your game at the highest quality setting to see if devices support it.

Run a technical test (compile multiple shaders, execute them, check that the
output is as expected, etc).

Test Loop Manager

To help you integrate game loops and run them on your local devices, as well as
to help your Quality Assurance team run game loops on their devices, we provide
an open-source app called Test Loop Manager.

Install Test Loop Manager on your device, using the following command:

adb install testloopmanager.apk

Open the app (named "Test Loop Apps") on your phone or tablet; you
are presented with a list of apps on the device that contain game loops. If
you do not see your app here, check to ensure that your intent
filter matches the one described in
Launch the loop.

Select the game app that you want to test.

After clicking the game app's name from the list, select the scenarios
(i.e., the game loops) that your app implements. If your app implements
multiple loops, see
Multiple game loops.

Click Run test, and watch as the scenarios are run.

Note: The default timeout is 3 minutes, meaning any running scenarios are
terminated at that time and any pending scenarios are canceled. To change it,
tap the menu button in the top right, click Set timeout, and change the
timeout in the dialog that appears.

Close the game app

At the end of your test, you should close the app using:

JavaAndroid

KotlinAndroid

Generally the game loop test allows the UI framework to start the next loop.
If you do not close the app, the UI framework running your loops won't know that
the test has finished, and may terminate the app after some time has passed.

Run the game loop test in Test Lab

Go to Firebase console and create a project if you
do not already have one. To use Test Lab at no charge, but with limited daily
quota, use the Spark billing plan. To learn about using Test Lab with no
usage limits, see
Firebase Pricing Plans.

Note: Virtual devices are not recommended for use with the game loop test
because of they have lower graphics frame rates than physical devices.

Use the Firebase console

Use the following instructions to run a game loop test using the
Firebase console:

In the Firebase console, click Test Lab in the left navigation panel.

Click Run Your First Test (or Run a Test, if your project has
previously run a test).

Select Game Loop as the test type, and then click Continue.

Click Browse, and then browse to your app's .apk file.

(optional) To select a subset of available scenarios (game loops) do one
of the following:

To choose specific scenarios, enter a list
or range of scenario numbers in the Scenarios field.

To choose all scenarios with a specific label
applied, enter one or more scenario labels in the Labels field.

Optional features

This section describes how to use optional features, such as writing data to an
output file, supporting multiple game loops, and labeling related loops
so that you can easily test related loops in a single test matrix.

Write output data

Your game loop can write output to a file that is provided using the
launchIntent.getData() method. This output data can be displayed by
Test Lab on the test results page. For an example of a data output file, see
Game loop test output file example.

Test Lab follows best practices for sharing a file between apps described at
Sharing a File.
In your Activity’s onCreate() method, where you are getting the intent, you
can also check for that file using the following code:

Multiple game loops

You can also support multiple game loops in your game app. For example, if you
have multiple levels in your game, you might want to have one game loop to
launch each level instead of having one loop that iterates through all of them.

That way, if your app crashes on level 32 you can directly launch that game
loop to try and reproduce the crash and to test bug fixes.

For example if you have 5 game loops in your app, you only need to add one line
to your app's manifest file, inside of the <application> element:

<meta-data
android:name="com.google.test.loops"
android:value="5" />

Then, when you launch Test Loop Manager, can use a selection screen to select
which loop to launch. If you select multiple loops to launch, it launches each
loop in sequence after the preceding loop completes. With Test Lab, you
select which loops to launch.

The launch intent contains the target loop as an integer parameter, in
the range of 1 to the maximum number of loops supported.

Note: Loops are assigned an integer starting with 1, not 0.

You can read the extra from the intent in Java as shown in the following
example code:

KotlinAndroid

Then, you can pass this extra to your native C++ code to change the behavior of
your loop based on the resulting int value.

Label game loops

You can label your game loop with one or more scenario labels to make it easy
for your QA team to launch a set of related game loops (for example,
"all compatibility game loops"). To do this, you need to add a meta-data
line to your app's manifest file that is similar to the following example:

Note: You can select ranges and sets of loops to label, and you can also combine
these notations. For example, to apply a label to loops 1 to 5, 9 and 11 to 20
you can set android:value to "1-5,9,11-20".

You can create your own labels, and we also provide 4 predefined labels:

com.google.test.loops.player_experience: For loops used to
reproduce a real user's experience when playing the game. The goal of
testing with these loops is to find issues that a real user would face while
playing the game.

com.google.test.loops.gpu_compatibility: For loops used to test
GPU-related issues. The goal of testing with these loops is to execute GPU
code that might not run properly run in production, to expose issues with
hardware and drivers.

com.google.test.loops.compatibility: For loops used to test a
broad range of compatibility issues, including I/O issues and OpenSSL
issues.

com.google.test.loops.performance: For loops used to test the
performance of the device. For example, a game might run at the most complex
graphics settings to see how a new device behaves.

App licensing support

Test Lab supports apps that use the
App Licensing
service offered by Google Play. To successfully check licensing when testing
your app with Test Lab, you must publish your app to the production channel
in the Play store. To test your app in the alpha or beta channel using
Test Lab, remove the licensing check before uploading your app to
Test Lab.

Known issues

The Game loop test in Test Lab has the following known issues:

Limited support for the Khronos Vulkan API. Only the following
devices available in Test Lab support the Vulkan API: Samsung Galaxy S7
(API level 24), Google Pixel (API level 25).

Some crashes do not support backtraces. For example, some release builds may
suppress the output of the debuggerd process using
prctl(PR_SET_DUMPABLE, 0). To learn more, see
debuggerd.

API Level 19 is not currently supported due to file permission errors.

Game loop test output file example

You can use output data files formatted as shown in the example below to display
game loop test results in the Firebase console. Areas shown as /.../ can
contain any custom fields that you need, as long as they don't conflict with
with the names of other fields used in this file:

{
"name": "test name",
"start_timestamp": 0, // Timestamp of the test start (in us).
Can be absolute or relative
"driver_info": "...",
"frame_stats": [
{
"timestamp": 1200000, // Timestamp at which this section was written
It contains value regarding the period
start_timestamp(0) -> this timestamp (1200000 us)
"avg_frame_time": 15320, // Average time to render a frame in ns
"nb_swap": 52, // Number of frame rendered
"threads": [
{
"name": "physics",
"Avg_time": 8030 // Average time spent in this thread per frame in us
},
{
"name": "AI",
"Avg_time": 2030 // Average time spent in this thread per frame in us
}
],
/.../ // Any custom field you want (vertices display on the screen, nb units …)
},
{
// Next frame data here, same format as above
}
],
"loading_stats": [
{
"name": "assets_level_1",
"total_time": 7850, // in us
/.../
},
{
"name": "victory_screen",
"total_time": 554, // in us
/.../
}
],
/.../, // You can add custom fields here
}