Chromium uses source-based code coverage for clang-compiled languages such as C++. This documentation explains how to use Clang’s source-based coverage features in general.

In this document, we first introduce the code coverage infrastructure that continuously generates code coverage information for the whole codebase and for specific CLs in Gerrit. For the latter, refer to code_coverage_in_gerrit.md. We then present a script that can be used to locally generate code coverage reports with one command, and finally we provide a description of the process of producing these reports.

Coverage Infrastructure

There are 3 layers in the system:

Coverage Builders

The first layer is the LUCI builders that

build instrumented targets,

run the instrumented tests,

merge the results into single streams,

upload data to cloud storage.

There are two types of builder:

CI Builder

The code coverage CI Builders periodically build all the test targets and fuzzer targets for a given platform and instrument all available source files. Then save the coverage data to a dedicated storage bucket.

CQ Builder

The code coverage CQ builders instrument only the files changed for a given CL. More information about per-cl coverage info in this doc.

Coverage Service

The second layer in the system consists of an AppEngine application that consumes the coverage data from the builders above, structures it and stores it in cloud datastore. It then serves the information to the clients below.

Coverage Clients

In the last layer we currently have two clients that consume the service:

Coverage Dashboard

The coverage dashboard front end is hosted in the same application as the service above. It shows the full-code coverage reports with links to the builds that generated them, as well as per-directory and per-component aggregation, and can be drilled down to the single line of code level of detail.

Refer to the following screenshots:

Directory View

See coverage breakdown by directories (default landing page).

Component View

Use the view dropdown menu to switch between directory and component.

Source View

Click on a particular source file in one of the views above to see line-by-line coverage breakdown, and it's useful to identify:

The command above builds crypto_unittests and url_unittests targets and then runs them individually with their commands and arguments specified by the -c flag. For url_unittests, it only runs the test URLParser.PathURL. The coverage report is filtered to include only files and sub-directories under url/ and crypto/ directories.

Aside from automating the process, this script provides visualization features to view code coverage breakdown by directories and by components, similar to the views in the coverage dashboard above.

Workflow

This section presents the workflow of generating code coverage reports using two unit test targets in Chromium repo as an example: crypto_unittests and url_unittests, and the following diagram shows a step-by-step overview of the process.

Step 0 Download Tooling

Generating code coverage reports requires llvm-profdata and llvm-cov tools. Currently, these two tools are not part of Chromium’s Clang bundle, coverage script downloads and updates them automatically, you can also download the tools manually (tools link).

Step 1 Build

In Chromium, to compile code with coverage enabled, one needs to add use_clang_coverage=true and is_component_build=false GN flags to the args.gn file in the build output directory. Under the hood, they ensure -fprofile-instr-generate and -fcoverage-mapping flags are passed to the compiler.

Step 2 Create Raw Profiles

The next step is to run the instrumented binaries. When the program exits, it writes a raw profile for each process. Because Chromium runs tests in multiple processes, the number of processes spawned can be as many as a few hundred, resulting in the generation of a few hundred gigabytes’ raw profiles. To limit the number of raw profiles, %Nm pattern in LLVM_PROFILE_FILE environment variable is used to run tests in multi-process mode, where N is the number of raw profiles. With N = 4, the total size of the raw profiles are limited to a few gigabytes.

Step 3 Create Indexed Profile

Raw profiles must be indexed before generating code coverage reports, and this is done using the merge command of llvm-profdata tool, which merges multiple raw profiles (.profraw) and indexes them to create a single profile (.profdata).

At this point, all the raw profiles can be thrown away because their information is already contained in the indexed profile.

Contacts

Reporting problems

Mailing list

FAQ

Can I use is_component_build=true for code coverage build?

Yes, code coverage instrumentation works with both component and non-component builds. Component build is usually faster to compile, but can be up to several times slower to run with code coverage instrumentation. For more information, see crbug.com/831939.

I am getting some warnings while using the script, is that fine?

Usually this is not a critical issue, but in general we tend not to have any warnings. Please check the list of known issues, and if there is a similar bug, leave a comment with the command you run, the output you get, and Chromium revision you use. Otherwise, please file a new issue providing the same information.

How do crashes affect code coverage?

If a crash of any type occurs (e.g. Segmentation Fault or ASan error), the crashing process might not dump coverage information necessary to generate code coverage report. For single-process applications (e.g. fuzz targets), that means no coverage might be reported at all. For multi-process applications, the report might be incomplete. It is important to fix the crash first. If this is happening only in the coverage instrumented build, please file a bug.

Why is coverage for X not reported or unreasonably low, even though there is a test for X?

There are several reasons why coverage reports can be incomplete or incorrect:

A particular test is not used for code coverage report generation. Please file a bug.

A test may have a build failure or a runtime crash. Please check the build for that particular report (rightmost column on the coverage dashboard). If there is any failure, please upload a CL with the fix. If you can't fix it, feel free to file a bug.

A particular test may not be available on a particular platform. As of now, only reports generated on Linux and CrOS are available on the coverage dashboard.

Is coverage reported for the code executed inside the sandbox?

Not at the moment until crbug.com/842424 is resolved. We do not disable the sandbox when running the tests. However, if there are any other non-sandbox'ed tests for the same code, the coverage should be reported from those. For more information, see crbug.com/842424.