This document covers running and writing integration tests for Gaia apps — written in JavaScript and run via Marionette — and provides a detailed explanation of how the integration tests are run.

Running tests

This section looks at setting up your environment correctly to run the existing integration test suite. The prerequisite is a working clone of the gaia repository. Note that tests live with the rest of the apps code (in apps/my_app/test/marionette for example) and test files always end in _test.js. Shared code for tests lives under shared/test/integration.

Setup

Shortcut: If you don't want to mess with your own environment, you can try MozITP — this toolset will automatically setup a pre-configured Ubuntu VM and Gaia Integration Test environment, allowing you can start testing on Mulet or a real device in one click.

As of late November 2015, to setup your environment for running tests on the master branch you need:

Running all the tests

The following command runs all the integration tests for all the apps. It will also install the marionette-js-runner and associated Node.js modules if not already done.

make test-integration

Important: If you get the error:

npm ERR! 404 'gaia-marionette' is not in the npm registry.

Make sure you really have npm 2.0.

You will get to watch the tests pop up on your screen and you will probably also get to hear various sounds as they run. This can be distracting or possibly maddening. To run specific tests you can set the TEST_FILES environment variable. To do so in the bash shell (default on Mac OS X), use a command of the form:

TEST_FILES=<test> make test-integration

For example, you could run the day_view_test.js test in calendar app with the below command.

TEST_FILES=./apps/calendar/test/marionette/day_view_test.js make test-integration

If you would like to run more than one test, use a space-separated list of files within the TEST_FILES variable:

Invoking tests for a specific app

To run just the tests for a specific app, you can use the following form:

make test-integration APP=<APP>

For example, you could run all tests for the calendar app with

make test-integration APP=calendar

Running the tests in the background, quietly

You might want to run the tests in the background, meaning that you won't have to see the tests run, worry about them stealing your focus, or risk impacting the running tests. One solution is to use Xvfb:

xvfb-run make test-integration

You can also run Xephyr:

# In one terminal
Xephyr :1 -screen 500x700
# in another terminal
DISPLAY=:1 make test-integration

If you are using PulseAudio and want to keep the tests quiet, then you can specify PULSE_SERVER=":" to force an invalid server so no sound is output:

PULSE_SERVER=":" make test-integration

You can of course combine both:

PULSE_SERVER=":" xvfb-run make test-integration

Running tests without building a profile

If you would like to run tests without building a profile, use make test-integration-test:

PROFILE_FOLDER=profile-test make # generate profile directory in first time
make test-integration-test

Running tests with a custom B2G Desktop build

By default, the test harness downloads a prebuilt version of B2G Desktop and runs the tests with that.

Sometimes, you want to run the tests with a custom build of B2G Desktop instead (for example, to test a local Gecko patch). To do this, first build B2G Desktop as described here, and then run:

RUNTIME=<path to objdir of b2g desktop build>/dist/bin/b2g make test-integration

On OS X, you need to run the B2G Desktop app from within its bundle, so run:

RUNTIME=<path to objdir of b2g desktop build>/dist/B2G.app/Contents/MacOS/b2g make test-integration

Running tests on device

You can run tests on device by plugging in your phone and adding the BUILDAPP=device to the make command:

Troubleshooting

This section goes over some problems you may encounter while trying to run the integration tests.

Running on Linux

The test harness seems to assume that the "node" command resolves to the Node.js executable. On some Linux distributions, this is not the case, even when Node.js is installed. For example, on Debian Jessie, the "node" command is provided by the "Amateur Packet Radio Node program" package, and Node.js is available under the "nodejs" command instead.

Bug 1136831 is on file for this problem. In the meantime, a workaround is to symlink "node" to point to "nodejs". On Debian you can install the package nodejs-legacy to fix this:

apt-get install nodejs-legacy

If Marionette-JS Starts To Throw Errors

There are times where your test runs may leave B2G in a state that causes errors to be thrown if you try to run tests. We basically need to get B2G back to a clean state.

make really-clean # This will remove all downloaded parts of B2G
make reset-gaia # This will reset the device and reinstall FXOS

You may have to run this on the Aries to reset the device

adb root # Run root level commands on your device

Troubleshooting Mac OSX El Capitan

If you already have Firefox Nightly installed on your system, the tests may not run. The tests now run with Firefox Nightly, and pointing them to a custom version of the B2G simulator as described in the sections above will fail. Instead, you'll need to run the tests pointing to your installation of Firefox Nightly on your Mac. Run the following command to execute the tests:

Marionette is faster than my application!

An extremely common issue is that marionette performs UI actions much faster than the average user. Imagine, for instance, that we had an application with two simple views, each with a button to navigate to the other one. We might expect the following snippet of pseudocode to work:

Important: This is an example of what not to do. Most marionette tests that fail intermittently exhibit the following behavior.

This simple test tries to navigate from one view to the other, navigate back, and then make sure that the original view is displayed. This test will fail intermittently since there's a race condition between the application code — which renders the two views — and the test code — which makes the implicit (and incorrect) assumption that the UI is responding synchronously. Instead, we need to program our test to behave like a user. Users don't try to click things or read things before they are actually visible in the UI. Users "poll" the UI by looking at it until the thing they're waiting for shows up. Here's a better version of our test:

How should I structure my UI test libraries?

Most of the test code you will write won't be general purpose enough to warrant abstracting into a general-purpose plugin. For app-specific code, we recommend having a separate file/class/module for each view. We demonstrate this pattern in gaia's calendar test libraries. The views folder has a unique class for each Calendar app view. The general calendar object has methods to navigate between the different views. In each of the views, we "scope" methods to search for elements in the DOM contained within the view's root element.

Often the views will contain getter methods to read an element or data from the view and setter methods to write data to an <input> or <select> element. A good example is creating a calendar event, which involves calling setter methods on the edit event view and saving. It's also idiomatic to abstract multiple UI actions that form a single "logical" action into view methods. For instance, setting a collection of reminders for a calendar event involves tapping a select option for each reminder.

How the test runner works

This section provides a detailed review of how the test runner works, for those that are interested.

Note: All of the various ways the Marionette JavaScript integration tests are run happens via the same make test-integration path. Only parameters are changed.

What triggers the tests?

All of our jobs are run through the travisaction helper script from our travis-project-jobs repo. The key things to know about this script are that it looks in a special directory for a subdirectory matching whatever CI_ACTION it is given. The config file that controls this directory is gaia/travisaction.opts, which points at gaia/tests/travis_ci for us.

The custom NPM_REGISTRY may or may not continue to exist; our node_modules are now sourced from our gaia-node-modules repo.

The custom reporter mocha-tbpl-reporter just prints out TEST-START / TEST-PASS / TEST-UNEXPECTED-FAIL / TEST-END that the parsers consuming Buildbot output expect, instead of all of those cool checkmarks you get when using the (default) spec reporter we use locally and on Travis. Note that the time-stamps from the Buildbot output are much more realistic than what the spec reporter claims for test durations. Do not believe the spec reporter: it doesn't include your setup times!

bin/gaia-marionette is a wrapper around marionette-js-runner's bin/marionette-mocha and generally appears to be a place to add a few more defaults, duplicate a bunch of work already done in the Makefile, and generately be a place to cram stuff if you don't understand Makefiles / want to make things take longer to run by avoiding parallelization. The good news is that because of all this you can invoke it directly. The notable bits in here are:

It uses "find" to locate all potential tests. If APP is in the environment, it only looks for tests under a given app. It will find blacklisted test files, but these are filtered out by marionette-mocha.

gaia/shared/test/integration/setup.js is specified as the first script to run when we spin up the mocha runtime. This currently is just a place to require and add plugins for use across all app tests.

gaia/shared/test/integration/profile.js is specified as the base configuration for all marionette integration tests' Gecko profiles. Adding things to this file will cause all tests to have certain prefs set, settings set, and/or apps installed. You almost certainly should not add things to this file.

The ParentRunner instantiates a ChildRunner. The ParentRunner is boring and its name is somewhat misleading. Both the ParentRunner and ChildRunner live entirely in the same "runner" process. There is currently only ever one ChildRunner.

Now things get somewhat complicated, so let's take a second to get an overview of all the processes that can be active and how they work:

It forks the mocha / bin/_mocha node.js script to be the "mocha test" (sub-)process.

This is done using the node.js child_process.fork mechanism. This allows the "runner" process to send messages to the child and receive message events in return.

It gets bossed around by the "mocha test" process. Literally. The messages get received and converted into lookups on the ChildRunner object with apply() then called and a callback generated that will marshal the result back down to the client. This means that the runner process can and is told to:

This just loads and configures a bunch of marionette plugins to run during every step.

All of the test files that bin/marionette-mocha was made aware of, that made it through the manifest's blacklists/whitelists.

See below for details on the execution model and how a test run actually works.

The "host" process(es). These are B2G Desktop/Firefox/(magic adb control wrapper) instances.

These are Gecko processes, potentially a hierarchy of processes now that OOP is in play.

They are communicated with via the Marionette protocol.

"server" processes: These are test/fake servers spun up by various tests. These currently all seem to be spun up by the "mocha test" processes directly, but it's conceivable the "runner" process could also spin some up. Some known examples:

there are duplicates of this implementation in browser and homescreen for some reason, likely historical.

The e-mail app has fake IMAP/POP3/SMTP servers that it shares with its back-end implementation. These live in the mail-fakeservers repo. The fake-servers actually run in an XPCShell Gecko-style instance that initially communicates via a low-level IPC interface using json-wire-protocol to bootstrap equivalence with the back-end tests' mechanism and then afterwards with a synchronous HTTP REST interface for all the e-mail domain work. It's due for some cleanup.

What is in my global scope in my test file?

Marionette: this comes from lib/runtime.js. It is a function that's a wrapper around mocha's TDD suite() method. Source is in lib/runtime/marionette.js. It has the following notable properties annotated onto it:

client(stuffToAddToTheProfile, driver that defaults to the synchronous Marionette API): Bound to HostManager.createHost from lib/runtime/hostmanager.js.

plugin(name, module): This is a bound function that invokes HostManager.addPlugin from lib/runtime/hostmanager.js. It adds the plugin for the duration of your test suite.

The TDD interface pokes setup/teardown/suiteSetup/suiteTeardown/suite/test into the global namespace.

What are marionette plugins?

Marionette plugins are node.js modules which extend the native abilities of the js marionette client.

This runs checkGlobals() to make sure you didn't clobber anything unexpected into the global state. It fails your test if you did.

emits afterEach, which is what teardown maps to.

emits afterAll, which is what suiteTeardown maps to.

emits suite end.

emits end.

What is the life-cycle of the Gecko processes?

For your test suite (aka each top-level marionette('description', function() {...}) in your file), a new profile is created.

For each test (aka each test('description', function() {...}) in your marionette('description', ...)), the host gets restarted after each test().

The nitty gritty of this is that your call to marionette.client() invokes HostManager.createHost() in lib/runtime/hostmanager.js, which uses the mocha TDD infrastructure to decorate your suite with the following:

This builds a profile from scratch if it doesn't exist, otherwise the existing profile is reused but settings/etc. are clobbered to be how we want them.

The "runner" process waits for the host to start-up. It connects to the host with the async API and starts a session, then deletes it, and only generates the callback notification that will allow the "mocha test" process to know the host is ready.