Mutation Testing with Pitest – Part 2: SonarQube

This post follows on from the previous part. If you have not yet read it, we recommend you take a few minutes to do so now.

The previous part explained the basics of mutation testing, as well as the configuration and use of Pitest as a mutation testing framework for JVM-based programming languages. It was explained how to integrate Pitest into a project and use Maven to execute it, whereby Maven is called up by hand and the reports generated by Pitest are archived in target directory on the development machine. If, for instance, you want to run Pitest automatically every night and make the generated reports available to several project participants at a central location, this approach will quickly – i.e. immediately – reach its limits. SonarQube, a tool widely used for static code analysis, which is also used in projects at TRIOLOGY GmbH, presents a solution for this. In this post, we will explain how to run Pitest as part of a Jenkins Pipeline and transfer the reports to SonarQube. At this point, basic knowledge of SonarQube as well as Jenkins Pipelines is assumed. If this is not the case, we recommend you read this blog post by Josha on Statistical Code Analysis with SonarQube as well as the Jenkins Pipeline documentation. It’s also a good idea to read the blog post by Johannes on code completion for Jenkinsfiles in IntelliJ IDEA.

Setup

The purpose of this post is not to explain how to set up Jenkins or SonarQube; it is therefore assumed that you have both a Jenkins CI-Server as well as a SonarQube instance and that they are already configured. You can easily create such a setup by using Cloudogu, for instance.

In order to integrate Pitest reports in SonarQube, Jenkins is not necessarily required; however, for the sake of simplicity and practicality, this post is concerned with the interaction of Jenkins, SonarQube and Pitest. Spring PetClinic is used as an example project. We assume that the project has already created a basic Jenkinsfile to check it out from the version management, compile, run unit tests, measure code coverage, and generate SonarQube reports. The Jenkinsfile contains the following:

The variables for the SonarQube URL, the user name and the password must be set according to the respective setup. In general, however, you should avoid writing secrets such as passwords in the Jenkinsfile. The credentials plugin for Jenkins, for example, can help with this. Apart from the Jenkinsfile, this is the unmodified PetClinic from the GitHub repository.

Ready, set, go!

Now that the foundations have been laid, it’s time to get to work.

The Pitest plugin for Maven The configuration of the Pitest plugin for Maven differs only slightly from the minimal configuration in the pom.xml that was at the end of the first article:

Without further configuration, the plugin now generates its reports in HTML format, which the Pitest-SonarQube plugin (more about this later) cannot use. Thus, the existing configuration needs to be extended by changing the output format to XML:

There is no need to configure more in the pom.xml, unless you want to exclude individual classes or tests from the mutation analysis. More about that can be found in the plugin documentation.

The SonarQube Pitest plugin Now that the reports are in a machine-readable format, SonarQube needs to be configured so it can also use the data. SonarQube can’t do anything with the mutation test reports innately; luckily, there is a community plugin that can generate SonarQube issues from the XML reports.

The easiest way to install this plugin is via the SonarQube Update Center, which can be found under ‘Administration’ -> ‘System’ -> ‘Update Center’:

After selecting the plugin and clicking on ‘Install,’ the plugin will download and install. After the installation is complete, SonarQube must be restarted by clicking on the corresponding button to activate the changes.

Important: Activate SonarQube rule After the installation, it is important to activate the SonarQube rule ‘Survived Mutant.’ If you forget to do this, no issues for surviving mutations will be created. The rule must be added to the quality profile used. This can be done by searching for ‘Survived Mutant’ in the ‘Rules’ tab and then clicking on ‘Activate.’ After selecting the quality profile (in our example, ‘Sonar Way’), the rule is activated.

At this point, the ‘Rule Severity’ can also be set. The standard setting assesses killed mutations as ‘Major’ issues. SonarQube is now completely configured.

A Pitest widget can also be added to the custom dashboard, if desired. This can be done on the project overview page by clicking on ‘Dashboards’ -> ‘Custom Dashboard’ -> ‘Configure Widgets.’ The widget can be found under the name ‘Pitest Reports.’

On the home stretch or: Jenkins

Theoretically, it is already possible to import the Pitest reports in SonarQube. For instance, here is the manual procedure:

Performing the unit test

Before the mutation tests can be performed, the unit test must be executed once. This can be done by entering mvn test.

Performing the mutation tests

As already explained in the first part, the mutation tests are executed by entering org.pitest:pitest-maven:mutationCoverage.

Pushing the reports to the Pitest-SonarQube plugin

This is a new step. Use the command mvn org.codehaus.mojo:sonar-maven-plugin:3.2:sonar -Dsonar.pitest.mode=reuseReport to call up the SonarQube-Maven plugin and push the reports generated by Pitest to SonarQube.

Of course, you wouldn’t want to do this manually on a regular basis. So we expand the Jenkinsfile accordingly:

These modifications automatically run the mutation tests after the unit tests and automatically transmit the resulting reports to SonarQube in the ‘SonarQube’ stage.

The duration of the Jenkins build depends, among other things, on the size of the project as well as the number of tests. An execution of the Pipeline described above might look like this:

Results

After the successful completion of the Jenkins build, SonarQube now determines surviving mutations as issues and displays them accordingly. An excerpt of the issues for the PetClinic project may look like this:

If it was activated, the Pitest widget will now be filled with numbers:

Of all mutations (111), 76.6% or 85 were killed. Of those, 30 were killed by the unit tests and 55 by timeouts. Timeouts occur when a mutation causes an infinite loop, for instance. After a configurable amount of time expires without the test failing, Pitest marks such a mutation as ‘killed by timeout.’

Of the 26 surviving mutations, 16 were not recognized by unit tests. An additional 10 mutations were found in places not covered by the unit tests and could therefore not be recognized at all (‘10 non covered mutations’).

As for the rate of killed mutations – the 76.6% from our example – there is no universally accepted best value as the gold standard. Certainly, the highest possible mutation coverage is desirable, but it must always be evaluated in relation to the effort required to obtain it.

Code If you would like to carry out our example yourself, you can clone the PetClinic project including the Jenkinsfile modification and the pom.xml from our GitHub repository.

Conclusion

The SonarQube-Pitest plugin makes it amazingly simple to integrate Pitest in SonarQube. Even automatic execution is no problem thanks to Jenkins. It is therefore hard to find a reason to not run mutation tests every night, for instance. Only the interpretation of the results is less simple, due to the lack of unambiguousness. Surviving mutations should always be checked by at least one actual person to see if they actually provide evidence of inadequate unit tests. This is something that cannot yet be automated.