Integrating Android project with Codecov

If you always search for ways to increase code quality and would like to encourage everyone in your
project to keep high test code coverage all the time, then Codecov may be another
step in your journey.

You can think of it as an automated code reviewer which constantly reminds you what you can
do to improve your test code coverage. It can be set up to perform build-failing checks on your code
changes if they would degrade code coverage or even suggest which files need to be covered by tests.

This article shows how to set up an open source Android project integration with Codecov as a part
of a Continuous Integration process with GitHub as a repository host and
Travis CI as a build server.

Generating code coverage reports

In order to get code coverage reports you have to gather test execution data first. This process may
differ between a Java module, an Android library project or an actual Android application project.
Following paragraphs describe the necessary configuration based on Gradle build
system.

Unit tests

To generate unit tests’ code coverage reports for an Android application or library
project some additional work may be needed. Currently,
Android Plugin for Gradle
does not generate Gradle report tasks out of the box. One solution is to configure them manually or to use
jacoco-android-gradle-plugin
that configures code coverage report tasks automatically:

To generate code coverage reports for all variants of an application or library you just run
jacocoTestReport Gradle task:

$ gradle jacocoTestReport

Now, you can browse to build/reports/jacoco/<variant_name> directory and see some nicely generated
HTML reports. By default, the plugin also provides an XML output which will be used by Codecov later
on.

The plugin also contains some predefined exclusion filters for generated classes so they do not end
up in code coverage reports. If there is still a need to provide some additional exclusion filters
it is possible to add them in the plugin extension block:

jacocoAndroidUnitTestReport{jacocoExcludes+=['**/AutoValue*.class']}

Instrumentation tests

Apart from testing on a local JVM, Android supports running tests on a real device or an emulator using
Instrumentation.
In case of instrumentation tests it is a little bit less hassle. You do not need to create code
coverage report tasks manually. They are already created by the Android Gradle plugin. All you have
to do is set testCoverageEnabled to true in your build type configuration, e.g.:

There is just a slight problem if you want to gather code coverage execution data on a real device
as there are some devices which do not allow that out of the box. Still, if you are using an emulated
environment it works rather well.

Unit tests in pure Java project

Fortunately, in case of a pure Java project close to no configuration is required in order to get
code coverage reports. You just apply the jacoco plugin with XML output enabled:

applyplugin:'jacoco'jacocoTestReport{reports{xml.enabledtrue}}

and you are ready to launch test and jacocoTestReport tasks.

Pushing reports to Codecov as a CI process step

The real power of integrating code coverage reports with Codecov comes from setting it as a
part of the CI process. This section describes such integration with an open source project
hosted at GitHub and built by Travis CI.

Whenever developers make a pull request to the project it would be great for them to see reports
on how merging their changes may affect the code coverage.

To achieve this with Travis CI a proper .travis.yml located in the project root sets the
environment to build an Android project.

The actual report upload is done by the bash <(curl -s https://codecov.io/bash) command.
Note that this configuration handles both regular unit tests run on the host machine and also
instrumentation tests run on the emulator. You should also remember to adjust the Android SDK,
build tools and emulator SDK version settings to suit your project’s configuration.

You should also go to your Travis profile page to enable
the trigger for the project’s repository so Travis builds it whenever a code change is introduced.

Now, creating a pull request at the project GitHub page should trigger a build. When it succeeds,
the codecov-io bot leaves a comment on the pull request with the code coverage information and
a link to the project page at Codecov for more details.

Fortunately, Codecov supports uploading multiple report files without the hassle of merging them
into one beforehand. If you have a multi-module project or a multi-variant Android project you do
not have to take any additional steps here.

Summary

There are other solutions that can be used to analyze code coverage reports, provide suggestions
and perform checks if the change would degrade the project quality.

Coveralls for example offers similar functionality. Currently, the available
coveralls-gradle-plugin expects the user to merge
test coverage reports into a single report in a multi-module Gradle project before publishing them
to Coveralls.

Probably, the most feature-rich solution is SonarQube™ as it performs
static code analysis alongside code coverage reports. It is also possible to feed it with integration
test results separately from unit test so it is easier to distinguish them in the end report.
The only downside is that it requires self-hosting.

In case of open source and cloud-hosted projects using Codecov may be the most elegant solution
at the time of writing this article. If you want to enhance your CI process to get the most out
of the code coverage reports then integrating with Codecov may probably be the way to go.

Feel free to comment and ask questions. I will be more than happy to answer them.