Telemetry tests

We use Telemetry benchmarks for testing and analyzing energy usage. See code concepts for details on benchmarks, measurements, and page sets.

Measurements

On desktop, the tab_switching measurement opens a group of tabs and measures energy after they are all loaded. All except the last tab are backgrounded. You can use the benchmark with a blank page in the foreground to understand how background tabs behave, or with a regular page in the foreground to focus on the foreground tab.

On android, our chromium.perf tests and most of our development is on chrome shell, which does not support tabs. So we use the power test instead. It will load each page in the set, with a 20 second pause, a scroll, and another 20 second pause in between each load.

Page sets

It might be a good idea to record a custom page set if you’re working on a very specific issue, for example below-the-fold animated gif. But we generally use the below page sets for most of our testing and debugging:

Desktop:

typical_25 - This is a set of 25 web pages designed to represent general web browsing.

Metrics

Each platform has different techniques for measuring energy usage, and often there are multiple techniques (hardware devices, software estimation). Below are the names of the metrics, the platforms they are available on, and how they are gathered.

energy_consumption_mwh - This is the energy consumption in milliwatt hours. In general, this is a software estimation. You can see all the methods we have for measuring energy consumption in the power_monitor directory in telemetry, but here is a quick run-down of how the bots on the waterfall calculate this:

Windows and Linux use MSR on intel processors with sandybridge or later.

application_energy_consumption_mwh - This is derived by measuring energy_consumption_mwh during a quiescent period before running chrome, then subtracting that from the energy_consumption_mwh used during the test run. The goal is to find the amount of energy used by chrome and ignore the steady state energy used by the OS and background processes. However, if there is too much noise this number can be off.

idle_wakeups_total - This is the total number of idle wakeups across all processes. It’s currently only available on Mac through powermetrics, but we are working on Linux (bug 422170) and Android (bug 422967).This number is important because it’s a proxy for energy consumption, and it figures heavily in the Energy Impact score in OS X. See the Maximizing Battery Life on OS X talk for more info about how reducing idle wakeups improves battery life.

board_temperature - Temperature in degrees celsius. Available through MSR on Windows/Linux (intel sandybridge and later). Also available on Android devices running L release+.

Developing/Testing Locally

Choose or record a telemetry page set that matches the use case you want to test.

Run the correct telemetry benchmark for your platform (tab_switching for desktop, acceptance test for android). Use the --profiler=trace and --pageset-repeat=1 flags; this will output a trace and only repeat the test once. Example:tools/perf/run_benchmark tab_switching.five_blank_pages \ --browser=release --profiler=trace --pageset-repeat=1

Telemetry outputs a trace file. You’ll see the file name in stdout. Open it up in about:tracing. Look for places where:

Chrome is doing unexpected work. For example, a timer that wakes every 1s on a blank page should not be needed. Removing unneeded timers is a great way to reduce idle wakups.

Work is happening too frequently and could be coalesced. Could two IPCs be grouped into one? Doing the same amount of work with fewer IPCs reduces idle wakeups.

Comparing traces between two builds that have different energy usage can also be helpful. You can use the --browser argument in telemetry to easily run the test on your stable, beta, or dev channel and compare that with a local build (--browser=release or --browser=debug).

Verifying Results on Perf Bots

Energy measurements are very sensitive to noise. Sometimes you make a change locally that drives down idle wakeups and you believe will reduce energy consumption, but the energy numbers you get are just noise. Sometimes you don’t have access to the OS or hardware you want to test. The chromium.perf and tryserver.chromium.perf bots can help!