Description

It would be excellent to have the integration tests for a component included along with the unit tests in the overall code coverage report for a module.

For example, a typical Maven 2 project may be

mycomponent
mycomponent-integration-tests

Sonar could generate the combined report for the mycomponent module in one of two ways:

1. Run Cobertura on mycomponent unit tests with Cobertura - this will produce a cobertura.ser file in the mycomponent/target/cobertura directory.
2. Run the mycomponent-integration-tests integration tests with Cobertura- this will produce a cobertura.ser file in the mycomponent-integration-tests/target/cobertura directory.
3. Amalgamate the two cobertura.ser files
4.Generate the code coverage report for the mycomponent module from the amalgamated cobertura.ser file.

OR

1. Run Cobertura on mycomponent unit tests with Cobertura - this will produce a cobertura.ser file in the mycomponent/target/cobertura directory.
2. Run the mycomponent-integration-tests integration tests with Cobertura, having the results being written to the cobertura.ser file in mycomponent/target/cobertura
3. Generate the code coverage report for the mycomponent module from the cobertura.ser in mycomponent/target/cobertura.

I'm pleased you've decided to add this feature into version 1.6 of Sonar. I think this, in itself, will be an excellent addition to the functionality, particularly given that this issue is only included as part of a paid product (i.e. Clover). I haven't tested getting entire coverage with Emma through the use of multiple metadata files.

Mik Quinlan
added a comment - 09/Feb/09 5:20 AM Hi Simon
Thanks very much for the links. I will check them out today.
I'm pleased you've decided to add this feature into version 1.6 of Sonar. I think this, in itself, will be an excellent addition to the functionality, particularly given that this issue is only included as part of a paid product (i.e. Clover). I haven't tested getting entire coverage with Emma through the use of multiple metadata files.
I look forward to this feature being released!
Mik

For those of you watching this issue, since submitting this improvement request, I've partially solved the problem. It does, however, require purchasing Clover. Sonar's clover plugin executes in the test and integration-test phase of Maven and produces a combined code coverage report. The limitation is that the unit tests for a module produce code coverage for that module only. So in a multi-module project, if you have a separate module called xxx-acceptance-tests that are for blackbox automated acceptance tests, you will not get code coverage for, say, your web module or service modules.

To get this code coverage to work and have tests run in the right phases, I have done the following:

1. Make all unit tests have the suffix UnitTest.java
2. Make all integration tests have the suffix IntegrationTest.java
3. Configure the Surefire plugin to run those tests in the right phase (*UnitTest.java in the test phase, *IntegrationTest.java in the integration-test phase) by doing the following:

Mik Quinlan
added a comment - 02/Jun/10 3:20 AM - edited For those of you watching this issue, since submitting this improvement request, I've partially solved the problem. It does, however, require purchasing Clover. Sonar's clover plugin executes in the test and integration-test phase of Maven and produces a combined code coverage report. The limitation is that the unit tests for a module produce code coverage for that module only. So in a multi-module project, if you have a separate module called xxx-acceptance-tests that are for blackbox automated acceptance tests, you will not get code coverage for, say, your web module or service modules.
To get this code coverage to work and have tests run in the right phases, I have done the following:
1. Make all unit tests have the suffix UnitTest.java
2. Make all integration tests have the suffix IntegrationTest.java
3. Configure the Surefire plugin to run those tests in the right phase (*UnitTest.java in the test phase, *IntegrationTest.java in the integration-test phase) by doing the following:
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude>**/*IntegrationTest.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>test</goal>
</goals>
<phase>integration-test</phase>
<configuration>
<excludes>
<exclude>none</exclude>
</excludes>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
Hope that is of help.
Cobertura, of course, does not do combine the tests from the test phase and the integration-test phase hence the existence of this improvement.
This issue should be kept open for those of us seeking a solution that does not require non-free software.

Today i have tried to enable clover in sonar, but i think the aggragation is not called. In Atlassian Clover documentation it is described "The command clover2:aggregate goal is used for merging coverage data generated by multi-module projects".

Conny Kreyssel
added a comment - 04/Jun/10 3:00 AM Thanks, Mik.
Today i have tried to enable clover in sonar, but i think the aggragation is not called. In Atlassian Clover documentation it is described "The command clover2:aggregate goal is used for merging coverage data generated by multi-module projects".
See http://confluence.atlassian.com/display/CLOVER/Clover-for-Maven+2+Installation+Guide
But sonar doesnt call in my case the clover goal?
Any hints?

You're right Conny, the Sonar Clover plugin doesn't call the aggregate goal. It's very simple do this but then far more complex to change the coverage information on a maven module which has already been analysed by Sonar.

Freddy Mallet
added a comment - 04/Jun/10 5:26 AM You're right Conny, the Sonar Clover plugin doesn't call the aggregate goal. It's very simple do this but then far more complex to change the coverage information on a maven module which has already been analysed by Sonar.

Connie - yes, I have found the same thing. In fact, I have a web service multi-module project that has an acceptance-tests module. If I use clover2:aggregate outside of Sonar with Clover I get 90% code coverage. Within Sonar, only 42%.

I think it's important that the code coverage from black box acceptance tests is also included.

Perhaps we should open a new issue to have the Sonar and Clover combination also have the clover2:aggregate feature? Currently, Sonar with Clover has less functionality than Clover on its own.

I love Sonar, BTW. It's made my life very easy. I always seem to have an issue with the code coverage for some reason, though, and that, for me, is the most important feature!

Mik Quinlan
added a comment - 04/Jun/10 10:50 AM Connie - yes, I have found the same thing. In fact, I have a web service multi-module project that has an acceptance-tests module. If I use clover2:aggregate outside of Sonar with Clover I get 90% code coverage. Within Sonar, only 42%.
I think it's important that the code coverage from black box acceptance tests is also included.
Perhaps we should open a new issue to have the Sonar and Clover combination also have the clover2:aggregate feature? Currently, Sonar with Clover has less functionality than Clover on its own.
I love Sonar, BTW. It's made my life very easy. I always seem to have an issue with the code coverage for some reason, though, and that, for me, is the most important feature!

Conny Kreyssel
added a comment - 04/Jun/10 3:30 PM Thanks, Mik.
Same kind of project here. We have Hibernate domain classes in a separate module and this module has 0% test coverage, but its fully tested in integration tests.
@Freddy
Can you more explain the problem to add all module coverage results in sonar?

Connie - I'll create the new issue in the next couple of days. As a side, the instructions I gave, above, negate the need for integration tests in a separate module. You can put them in the same module and get the entire code coverage. This would solve your issue.

Freddy - yes, I'd be interested to know what the issue is. Perhaps it's a design issue and the Sonar metrics are collated as each module is built and analysed as opposed to storing the data in a temporary area and reading it all at the end of the analysis (which would allow aggregate coverage)? I'm sure it's not as simple as that so your input is appreciated.

Mik Quinlan
added a comment - 04/Jun/10 5:20 PM Connie - I'll create the new issue in the next couple of days. As a side, the instructions I gave, above, negate the need for integration tests in a separate module. You can put them in the same module and get the entire code coverage. This would solve your issue.
Freddy - yes, I'd be interested to know what the issue is. Perhaps it's a design issue and the Sonar metrics are collated as each module is built and analysed as opposed to storing the data in a temporary area and reading it all at the end of the analysis (which would allow aggregate coverage)? I'm sure it's not as simple as that so your input is appreciated.

Mik, your understanding is perfectly correct : once a module has been analyzed all measures at package, file, class and method levels are dumped in the DB and deleted from memory to prevent using too much memory.
Let's say for instance that a Maven project contains two modules : module A and module B :

Module A contains somes sources

Module B depends on Module A and contains some integration tests

Sonar starts to analyse Module A and store all measure in the DB. Then it analyses Module B, but as all resources from Module A have been purged from Memory there is no way (at that time) to change any measures on those resources (in fact, there is no way to know that a given resource is part of Module A).

Freddy Mallet
added a comment - 05/Jun/10 10:03 AM Mik, your understanding is perfectly correct : once a module has been analyzed all measures at package, file, class and method levels are dumped in the DB and deleted from memory to prevent using too much memory.
Let's say for instance that a Maven project contains two modules : module A and module B :
Module A contains somes sources
Module B depends on Module A and contains some integration tests
Sonar starts to analyse Module A and store all measure in the DB. Then it analyses Module B, but as all resources from Module A have been purged from Memory there is no way (at that time) to change any measures on those resources (in fact, there is no way to know that a given resource is part of Module A).

My project uses the surefire-setup as described above by Mik: integration tests are located in the same module as unit tests, but run in the "integration-test" phase. Confusingly, Sonar picks up the integration test results, but does not include the integration tests in the coverage report.

We use Cobertura, not Clover. I believe, with the plain Cobertura maven plug-in it would be possible to merge coverage data from two runs. Do you have any plans to support this scenario in Sonar as well (either aggregate coverage data or side-by-side with unit tests)?

Dennis Homann
added a comment - 17/Aug/10 12:44 PM My project uses the surefire-setup as described above by Mik: integration tests are located in the same module as unit tests, but run in the "integration-test" phase. Confusingly, Sonar picks up the integration test results, but does not include the integration tests in the coverage report.
We use Cobertura, not Clover. I believe, with the plain Cobertura maven plug-in it would be possible to merge coverage data from two runs. Do you have any plans to support this scenario in Sonar as well (either aggregate coverage data or side-by-side with unit tests)?

Same here, we have convention that integration tests sit in the same module but in a different package (starts with itest). They are triggered by passing a environment variable -Dskip.integration.test=false. These tests are run in 'integration-test' phase.

manuel aldana
added a comment - 07/Sep/10 9:22 AM Same here, we have convention that integration tests sit in the same module but in a different package (starts with itest). They are triggered by passing a environment variable -Dskip.integration.test=false. These tests are run in 'integration-test' phase.
We use clover.

elmandour omar
added a comment - 07/Jul/11 11:05 AM - edited As Evgeny Mandrikov mentionned, Jacoco made the trick for me & is plugin agnostic.
I have use it on a real entreprise & multi-module maven projects :
A-core
B-dao
C-web
Each of them have unit tests and C-web have integration test when the war is deployed.
Those integration-test are composed of Jwebunit and SoapUi.
For each module let say we have a 10% coverage.
But obviously C is using indirectly A & B so it will be nice to report the coverage done during the integration-test to the respective modules (A & B)
The trick is to not seperate unit & integration test and generate the jacoco.exec in one place.
Each module will append his result to the same file as long as the tests are run sequentially.
So the sonar maven plugin will analyse 3 times the same file for each modules.
And after, the coverage results could be A(30%), B(30%) and C (10% unchange of course)
Of course if you are really interested to seperate IT report coverage and Unit report coverage, use the same trick with two files for each need.
The difficult part is to agregate report file with tcp stuff, produced in a distant J2ee server ;(.
For this purpose of merge, there is a ant task to merge files.
Omar

Reinhold Erler
added a comment - 09/Feb/12 9:38 AM Hi Omar,
Your solution is very interesting! Please could you tell us your steps in more detail?
Did you:
mvn sonar:sonar on A-core
mvn sonar:sonar on B-dao
ant task for merging the 2 jacoco.execs
mvn sonar:sonar on C-web
And how did you manage to not overwrite the merged file when you called C-web?
Thank you very much for clarification!
statler

I have a bit of a different scenario from the ones described above, we have a c++ code base and use Bullseye c++ coverage and before actually running sonar all the data (unit test and integration test coverage) is available.

So when running sonar, both coverage data is feed into sonar in two different xml reports and finally when looking at source code tab for a project than we see both coverage in separate lists.

This is fine, but it would be nice that what ever solution found for feeding data into sonar, when looking into the source code tab we could see have a Checkbox that merges both.

Jorge Costa
added a comment - 18/Jun/12 11:11 AM Hi,
I have a bit of a different scenario from the ones described above, we have a c++ code base and use Bullseye c++ coverage and before actually running sonar all the data (unit test and integration test coverage) is available.
So when running sonar, both coverage data is feed into sonar in two different xml reports and finally when looking at source code tab for a project than we see both coverage in separate lists.
This is fine, but it would be nice that what ever solution found for feeding data into sonar, when looking into the source code tab we could see have a Checkbox that merges both.
Hope you get this one there.
Thanks
JC

I'm closing this ticket as in fact it duplicates SONAR-2804. This ability to get and play with three different code coverage : unit tests, integration tests, overall code coverage will be available only when using the Sonar Jacoco plugin.

Freddy Mallet
added a comment - 09/Oct/12 10:38 AM I'm closing this ticket as in fact it duplicates SONAR-2804 . This ability to get and play with three different code coverage : unit tests, integration tests, overall code coverage will be available only when using the Sonar Jacoco plugin.

Hi Daniel, This is coverage tool specific as soon as we want to also merge coverage by branch. Indeed the information contained into Sonar only allows to aggregate the coverage by lines of code. Moreover, computing a code coverage by integration tests is far more straightforward with Jacoco than it is with Cobertura. As in the first case you just need to plug the jacoco agent to the JVM.

Freddy Mallet
added a comment - 10/Oct/12 1:39 PM Hi Daniel, This is coverage tool specific as soon as we want to also merge coverage by branch. Indeed the information contained into Sonar only allows to aggregate the coverage by lines of code. Moreover, computing a code coverage by integration tests is far more straightforward with Jacoco than it is with Cobertura. As in the first case you just need to plug the jacoco agent to the JVM.