May we suggest...

Tracking our Android test coverage with Sonarqube

We recently shared about our transition from legacy code to clean architecture, where we’ve also begun to introduce testing to our project in the form of both Unit and Instrumentation tests. The introduction of these tests will help us to build a more stable product, boosting our confidence in the features that we ship.

But why are we talking about coverage?

Well, not only does this help us to measure how thoroughly tested our code is, but it’s also helping to act as a way to track our progress in the big refactor of our app. To help us with this we’re making use of two tools, these are:

Jacoco – a code coverage plugin that will generate reports that we can use for coverage analysis

Sonarqube – a platform that allows you to track metrics for projects such as technical debt, bugs, code coverage, etc.

The best way to learn about both of these is to set up both of the tools, run your tests and send the reports to Sonarqube – then you are free to explore your analyzed project from within Sonarqube.

The focus for today though is test coverage – and once you’ve sent the reports to Sonarqube, you’ll be able to see something a little like this:

This is great, right? Within Sonarqube you can explore packages to see the coverage for each, as well as coverage for each class inside of that package. Each Package/Class will give you a coverage level both in terms of a percentage and a color (red to green, which is great for glancing over).

Whether you already have tests in place or you’re just starting to add tests, introducing coverage is a great way of learning how well tested your code is and finding areas in which you can improve. We’re now going to take a look at how we set this up in our project. Follow along if you wish to start tracking the code coverage for your project ?

Setting up Jacoco for Unit Test Coverage

To begin with, we need to add the plugin at the top of our app-level build.gradle file:

1

apply plugin:'jacoco'

This plugin gives us access to the jacoco tool, allowing us to create reports for our tests when they are run.

Note: As of Gradle 2.2 we are required to add the following test options to our build.gradle file also, this enables coverage for tests found in our src/test/java directory.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

android{

...

testOptions{

unitTests.all{

jacoco{

includeNoLocationClasses=true

}

}

}

...

}

Finally, we need to configure the generation of our code coverage reports. We created the gradle task below to handle this:

reports – These are the different report file formats that we have enabled jacoco to create reports for

fileFilter – This is a collection of files that we wish to exclude from our coverage reports. This is extremely useful for removing any classes generated by Dagger, as well as any Test and framework classes from the test coverage.

debugTree – This is a variable for the location of the .class files for your project

mainSrc – This is a variable for the location of our projects main directory

sourceDirectories – Used by jacoco to collect java files for analysis, here we’re giving it the path stored in our mainSrc declaration

classDirectories – Used by jacoco to collect class files for analysis, here we’re giving it the path stored in our debugTree declaration

executionData – Used by jacoco to save the results of the report generation, this must have the .exec extension but you can decide the path

Now, if you run the gradle task we’ve just created, you’ll see that your unit tests will run and jacoco .exec file will have been generated at the path given for the executionData property.

Adding Coverage Reports for UI Tests

Now we have Jacoco creating coverage reports for our Unit Tests, we’re now going to add some coverage for our User Interface tests also. Luckily for us, there is already a gradle task called createDebugCoverageReport – this will run your UI tests on a device and then create a coverage report from the results. Because of this, we’re going to add it as a dependsOn task to the createTestReports task that we previously created:

Once the tests have run, a code coverage report will be generated and placed into a directory within your build directory, this will be something like:

1

your-app/build/outputs/code-coverage/connected/

This will be an .ec file with the name of the device that the tests have been run on and there will be a file generated for each of the devices that the tests are run on. For coverage, we only really care about reading a single coverage report as this coverage will be the same for each device. Because of this, we use the following commands to extract a list of files in the directory where the reports are generated and grab the first one’s filename. We store this name in the instrumentationFileName for use when we send the reports to sonar.

So now if you run this task you’ll find that your unit tests will run, followed by your Instrumentation Tests. Once both are complete (and passing!) you’ll notice that reports will have been generated in the directories that we stated.

Setting up Sonarqube

Sonarqube is fairly simple to setup and can be done so by following the instructions on this link. Setting up a local instance is great for trying out Sonarqube and working out if it fits your workflow. Then once you’re ready, it’s really simple to switch to a server instance as you just need to change the Sonarqube instance you’re pointing to once it’s been setup!

Configuring Sonarqube

Now we have our code coverage all configured, it’s time to configure Sonarqube so that we can view our coverage reports.

First, we need to add the Sonarqube plugin declaration to our build.gradle file:

1

2

3

plugins{

id"org.sonarqube"version"1.2"

}

Next, we need to define a collection of properties that will be used when we send our reports to Sonarqube. To do so, we need to place these declarations within our app-level build.gradle file:

itReportPath – The path pointing to your instrumentation test coverage report. You’ll notice here that we’re using the instrumentationFileName path that we declared earlier. This is the filename of the single instrumentation test coverage report that was generated from our createTestReports task

Once you’ve set all of these properties to their corresponding values, you’re all set to run the sonarqube task and send your coverage reports to your sonarqube instance ?

We’re now covered!

So now we have coverage reports generated for all of our tests and Sonarqube is configured we’re able to send our reports to sonar. We can easily do this by running the following command:

1

./gradlew clean sonarqube

However, we decided to extend on this command so that it depends on our createTestReports task. This way our tests will run, reports will be generated and then uploaded to sonar within a single task. To achieve this yourself, simply add the following to your app-level build.gradle file:

1

tasks.sonarqube.dependsOn createTestReports

Running the sonarqube command this time around will now run your tests and generate the reports before sending them to sonarqube ?

Note: Whilst choosing our instrumentation server setup, we put together a little script that will prompt you to run the tests locally and push to sonarqube if you choose to do so. You can check this out on our Github page here!

1

2

3

4

5

6

7

8

9

10

11

12

13

#!/bin/sh

# Show a dialog to ask if UI Tests should be run and results pushed to Sonarqube

I’m an Android Engineer and I love creating beautiful, clean and functional applications that help to make peoples lives easier – I love to be constantly learning and I enjoy writing about Android things / working on open-source projects over on GitHub. I have a bit of a love for Android TV which I sometimes talk about at conferences too! I first found my passion for Android whilst I was at university and have been working with it non-stop since – I find it an amazing platform. I love how it’s constantly evolving, it’s open-ness and the community around it!

I’m actively trying to get coverage data to work. Unfortunately, whenever I upload both the unit test coverage (.exec) file and the instrumentation test coverage (.ec) file, I get an error about duplicate IDs. A lot of searching has said that I need to “merge” the reports together, but I’ve had zero luck getting that to actually work in practice. Has anyone else had any luck here?

Manos Tsahakis

I had a problem when adding
property “sonar.java.binaries”, “build/intermediates/classes/dev/debug”

sonar added bugs that were not there, like not closing resources in finally.
Once I removed that, all errors went away.