Description

In my project, I have both unit tests ("test" phase) and integration tests ("integration-test" phase).
So far I could manage configuring maven-surefire-plugin and maven-surefire-report-plugin to execute both tests correctly and also generate 2 different reports.
Then I have added cobertura-maven-plugin to the reporting in order to get coverage but unfortunately only unit tests have their coverage reported (I know it because I have some classes which are only integration tested but are reported as 0% covered).
After trying to find information on the mailing lists, on the web and other existing resources, I could not find any hint on how to make this work.
It looks like cobertura-maven-plugin, by its current design, will never run integration-test to collect coverage, it seems to stop at the "test" phase.

Thus whenever a POM project has integration tests and uses cobertura-maven-plugin for coverage report, the generated reports are wrong, which is very misleading.

Actually, I was surprised not to find this issue already in JIRA.

Is there a chance this gets fixed soon? Or is there a usable workaround for this problem (besides switching to clover which I am not sure it would work better ) Did someone succeed in patching cobertura-maven-plugin to get the correct behavior?

Activity

The documentation for cobertura:cobertura flat out says that it runs the "test" phase prior to executing itself, so I assume this currently cannot be configured. There's of course much more complicated cases that people use the integration-test phase for, but for a lot of projects, it'd be enough if you could configure Cobertura to at least include or run integration-test phase instead. Email thread at http://www.mail-archive.com/users@maven.apache.org/msg78743.html.

Kalle Korhonen
added a comment - 10/Mar/08 2:36 PM The documentation for cobertura:cobertura flat out says that it runs the "test" phase prior to executing itself, so I assume this currently cannot be configured. There's of course much more complicated cases that people use the integration-test phase for, but for a lot of projects, it'd be enough if you could configure Cobertura to at least include or run integration-test phase instead. Email thread at http://www.mail-archive.com/users@maven.apache.org/msg78743.html .

Here's a patch, with new cobertura:cobertura-integration report mojo, which extends existing cobertura report mojo but instead of test phase executes verify phase to have pre-integration-test, integration-test, and post-integration-test phases to execute as well. To use it one should define in pom reporting section something like this:

This has been tested only on a trivial example project, and appears to be working well. I'm absolute beginner when it comes to mojo development, so this patch should be treated in that light, just as an suggestion on how this issue could be resolved.

The second patch adds one more new cobertura:report-only mojo, which executes only validate phase. Idea is that one should be able to use it together with cobertura:clean cobertura:instrument mojos, for fine-grained control over instrumentation process.

Stevo Slavic
added a comment - 03/Apr/09 4:31 AM Here's a patch, with new cobertura:cobertura-integration report mojo, which extends existing cobertura report mojo but instead of test phase executes verify phase to have pre-integration-test, integration-test, and post-integration-test phases to execute as well. To use it one should define in pom reporting section something like this:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<version>2.3-SNAPSHOT</version>
<reportSets>
<reportSet>
<reports>
<report>cobertura-integration</report>
</reports>
</reportSet>
</reportSets>
</plugin>
This has been tested only on a trivial example project, and appears to be working well. I'm absolute beginner when it comes to mojo development, so this patch should be treated in that light, just as an suggestion on how this issue could be resolved.
The second patch adds one more new cobertura:report-only mojo, which executes only validate phase. Idea is that one should be able to use it together with cobertura:clean cobertura:instrument mojos, for fine-grained control over instrumentation process.

I've tested and can confirm that both new Mojo's work well with relatively big multi-module projects.

Can someone else please try too?

Is it possible to get at least report-only mojo included in 2.3? With it, and with existing clean and instrument mojos, one can have more complete control over what and when gets instrumented, run tests and then get a coverage report only without rerunning of tests just for report. cobertura-integration mojo is not needed, but can be useful for simpler use cases, where with less configuration one can get coverage data for tests run in integration-test phase along with coverage data for tests in test phase.

Stevo Slavic
added a comment - 21/Apr/09 6:39 AM I've tested and can confirm that both new Mojo's work well with relatively big multi-module projects.
Can someone else please try too?
Is it possible to get at least report-only mojo included in 2.3? With it, and with existing clean and instrument mojos, one can have more complete control over what and when gets instrumented, run tests and then get a coverage report only without rerunning of tests just for report. cobertura-integration mojo is not needed, but can be useful for simpler use cases, where with less configuration one can get coverage data for tests run in integration-test phase along with coverage data for tests in test phase.

Tom Vaughan
added a comment - 05/Jun/09 1:24 PM Hi,
How/where do I go to get the 2.3-SNAPSHOT release? I don't see it on ibiblio, but I'd love to get that version to help test this function.
Thanks,
Tom

To test the cobertura:report-only mojo patch one can check out latest cobertura-maven-plugin sources from http://svn.codehaus.org/mojo/ under trunk/mojo/cobertura-maven-plugin, apply the patch, install the plugin snapshot in to your local repository, and then use it from your maven project.

Stevo Slavic
added a comment - 06/Jun/09 8:18 AM To test the cobertura:report-only mojo patch one can check out latest cobertura-maven-plugin sources from http://svn.codehaus.org/mojo/ under trunk/mojo/cobertura-maven-plugin, apply the patch, install the plugin snapshot in to your local repository, and then use it from your maven project.

Hi Stevo,
I've patched the 2.3 snapshot with your work, and it works on big pom tree (parent/module) configuration.
But how did you solved the "classesDirectory " problem when using : surefire+selenium+cobertura when executing test, deployment, it and site into an unique maven execution ?
Removing the "readOnly" flag from the war plugin slove partially the problem, but i think there is a much simplier solution to run correctly cobertura ....
Can you attach a pom sample and the associated maven command line you use please?

Fabrice Daugan
added a comment - 18/Jun/09 1:36 AM Hi Stevo,
I've patched the 2.3 snapshot with your work, and it works on big pom tree (parent/module) configuration.
But how did you solved the "classesDirectory " problem when using : surefire+selenium+cobertura when executing test, deployment, it and site into an unique maven execution ?
Removing the "readOnly" flag from the war plugin slove partially the problem, but i think there is a much simplier solution to run correctly cobertura ....
Can you attach a pom sample and the associated maven command line you use please?
Thanks

Wish "report-only" mojo was included in 2.3, hopefully it will get included in 2.4, together with this new one, "check-only" mojo (see attached cobertura-maven-plugin_check-only-mojo.patch). Current cobertura:check mojo is heavy, it includes running tests, so if one wants to run tests (e.g. with surefire), auto check coverage (cobertura:check), and get a coverage report in a site (using cobertura:cobertura), tests get run at least 3 times - having tests run twice (once on non instrumented code to check if code is ok, and once on instrumented code to get coverage) takes time but it's understandable, while having them run 3 times is way too much. Point is that cobertura maven plugin should be able to reuse coverage data for both check and for report generation tasks. With report-only and check-only this is feasible - with all these mojos at hand one could:

1) cobertura:clean
2) run the tests on non instrumented code and only if they pass execute...
3) cobertura:instrument
4) run the tests on instrumented code (this would generate coverage data)
5) cobertura:check-only, to check if coverage requirements are met, and only if they are execute...
6) cobertura:report-only, as part of project site generation

Stevo Slavic
added a comment - 01/Aug/09 7:55 PM Wish "report-only" mojo was included in 2.3, hopefully it will get included in 2.4, together with this new one, "check-only" mojo (see attached cobertura-maven-plugin_check-only-mojo.patch). Current cobertura:check mojo is heavy, it includes running tests, so if one wants to run tests (e.g. with surefire), auto check coverage (cobertura:check), and get a coverage report in a site (using cobertura:cobertura), tests get run at least 3 times - having tests run twice (once on non instrumented code to check if code is ok, and once on instrumented code to get coverage) takes time but it's understandable, while having them run 3 times is way too much. Point is that cobertura maven plugin should be able to reuse coverage data for both check and for report generation tasks. With report-only and check-only this is feasible - with all these mojos at hand one could:
1) cobertura:clean
2) run the tests on non instrumented code and only if they pass execute...
3) cobertura:instrument
4) run the tests on instrumented code (this would generate coverage data)
5) cobertura:check-only, to check if coverage requirements are met, and only if they are execute...
6) cobertura:report-only, as part of project site generation

Attaching patch with updated check-only and report-only mojos (cobertura-maven-plugin_check-only-and-report-only-mojos_with-IT), now with their integration test. Previous patches can be deleted/ignored.

I'm wondering if there is a danger in the new approach of running the individual steps. Let's say I attach the various goals to my default lifecycle. Isn't there a danger that a package step will package up the instrumented classes because the instrument goal does the following:

// Set the instrumented classes to be the new output directory (for other plugins to pick up)
project.getBuild().setOutputDirectory( instrumentedDirectory.getPath() );
System.setProperty( "project.build.outputDirectory", instrumentedDirectory.getPath() );

Michael Spaulding
added a comment - 29/Jan/10 3:56 PM I'm wondering if there is a danger in the new approach of running the individual steps. Let's say I attach the various goals to my default lifecycle. Isn't there a danger that a package step will package up the instrumented classes because the instrument goal does the following:
// Set the instrumented classes to be the new output directory (for other plugins to pick up)
project.getBuild().setOutputDirectory( instrumentedDirectory.getPath() );
System.setProperty( "project.build.outputDirectory", instrumentedDirectory.getPath() );

Candy Chiu
added a comment - 02/Feb/10 4:12 PM I tried the new cobertura:cobertura-integration mojo in 2.3-SNAPSHOT downloaded from http://snapshots.repository.codehaus.org . It didn't work on my project. All integration tests were skipped.

Hi, I wanted to be able to run my integration tests with cobertura. I added the http://snapshots.repository.codehaus.org to the repository and did an mvn clean. It then downloaded the jars.
I then ran mvn cobertura:cobertura-integration command but got an error that said the new mojo was not found.

Pasted below is the our put of the mvn clean command that shouws the jars being downloaded.

@Nikhil Almeida
Like the logs already tell you: the cobertura-integration goal doesn't exist. Not in cobertura-maven-plugin-2.3 or in the current trunk.
Although there are a lot of votes, this patch hasn't been added to the plugin yet.

Robert Scholte
added a comment - 22/Apr/10 2:45 PM @Nikhil Almeida
Like the logs already tell you: the cobertura-integration goal doesn't exist. Not in cobertura-maven-plugin-2.3 or in the current trunk.
Although there are a lot of votes, this patch hasn't been added to the plugin yet.

Benoit Xhenseval
added a comment - 26/Apr/10 2:46 PM Was there any reason why this patch was not applied to the new release done on April 25? Is any developer looking into this? Please, pretty please...

Navjeet
added a comment - 07/Sep/10 2:52 PM We are using version 2.4 and it seems to run integration tests but as unit tests (our integration tests are cactus based). So is it confirmed that v 2.4 of maven plugin won't run integration tests?

Rafael Mahnovetsky
added a comment - 19/Sep/10 9:36 PM I applied your patch and now I can run a cobertura report after integration test .. nice one!
I hope your patches get applied in the next release.
At least we can now have a choice if we want to include integration tests into our code coverage or not.

This issue is opened since 01/Mar/08. This is more than 2 years. There is a simple patch that works and it just adds two goals, so the plugin is backward compatible. Why this take so long to do ? May we have a response about when we can have this done or not. Because I'm about to do a fork of this plugin on google code with the possibility to do integration test coverage on war application which is really cool to see functional test coverage. Sorry for the tone, I really like your work
I attach the updated patch which work on latest 2.5-SNAPSHOT.

Amertum
added a comment - 25/Oct/10 4:54 AM This issue is opened since 01/Mar/08. This is more than 2 years. There is a simple patch that works and it just adds two goals, so the plugin is backward compatible. Why this take so long to do ? May we have a response about when we can have this done or not. Because I'm about to do a fork of this plugin on google code with the possibility to do integration test coverage on war application which is really cool to see functional test coverage. Sorry for the tone, I really like your work
I attach the updated patch which work on latest 2.5-SNAPSHOT.

I think the problem with this issue is that it only covers half of the problem. It will only work for classes executed by the m-failsafe-p, because it can use these instrumented classes. But you might want to do an integration-test on the generated jar/war/.. and that's not possible.
You don't want the (main) artifact to be instrumented, but you might want to have an instrumented version as well (with cobertura classifier) and use that one to do the integration-test.

To cover the first type of integration-tests I'll review the patches. I don't think it would harm to add a report-only goal to the plugin.

Robert Scholte
added a comment - 25/Oct/10 11:07 AM I think the problem with this issue is that it only covers half of the problem. It will only work for classes executed by the m-failsafe-p, because it can use these instrumented classes. But you might want to do an integration-test on the generated jar/war/.. and that's not possible.
You don't want the (main) artifact to be instrumented, but you might want to have an instrumented version as well (with cobertura classifier) and use that one to do the integration-test.
To cover the first type of integration-tests I'll review the patches. I don't think it would harm to add a report-only goal to the plugin.

I found a small trick that make integration-test works on instrumented jar/war/...

Obviously I use a maven profile that I only trigger during dev and on my ci server.
If my war application does not depend on other artifact, the code is quite simple. I use IT test (which are now understood by husdon ci) and I load my war (multiple times) using jetty during these tests. At the end, the cobertura results are flushed.

But when you have dependencies you want the coverage to go through. so I use a trick. I exclude my dependencies from the war build and instead I unpack the dependencies sources and attach them to the build. Maven compile and instrument these classes and we have the coverage we want

After reviewing this issue, I don't think the check-only/report-only is the right solution.
The cobertura-lifecycle has to be extended, and I think it should look something like this:

Before entering the test phase, the java-classes need to be instrumented for test
Before entering the integration-test phase, java-classes need to be instrumented for integration-test.
Now they both have their own .ser file, and during verifyall these should be checked and eventually reported.
I think this would result in less configuration and no need to misuse a report-only goal.

Robert Scholte
added a comment - 28/Oct/10 3:15 PM After reviewing this issue, I don't think the check-only/report-only is the right solution.
The cobertura-lifecycle has to be extended, and I think it should look something like this:
Before entering the test phase, the java-classes need to be instrumented for test
Before entering the integration-test phase, java-classes need to be instrumented for integration-test.
Now they both have their own .ser file, and during verify all these should be checked and eventually reported.
I think this would result in less configuration and no need to misuse a report-only goal.

I agree. So that way we can have two coverage reports, one for unit tests and the other for integration tests. And so, we can invoke mvn site and got results without running verify goal first, which is required with the patch.

Amertum
added a comment - 28/Oct/10 3:31 PM I agree. So that way we can have two coverage reports, one for unit tests and the other for integration tests. And so, we can invoke mvn site and got results without running verify goal first, which is required with the patch.

Miguel Almeida
added a comment - 14/Mar/12 5:24 AM Is there any news on this?
There are a couple of patches here, and there's even a fork at http://code.google.com/p/cobertura-it-maven-plugin/wiki/HowToUse to solve this issue.
Can any of it be used to implement this feature?

It seems like Jira is not the right tool to come to a solid/everything covering solution.
So I've started a confluence-page: http://docs.codehaus.org/display/MOJO/Cobertura+Maven+Plugin
This is quite a complex issue and is related to other cobertura-issues.
So let's first describe how the plugin should work.
Any contribution is appreciated.

Robert Scholte
added a comment - 14/Mar/12 4:26 PM It seems like Jira is not the right tool to come to a solid/everything covering solution.
So I've started a confluence-page: http://docs.codehaus.org/display/MOJO/Cobertura+Maven+Plugin
This is quite a complex issue and is related to other cobertura-issues.
So let's first describe how the plugin should work.
Any contribution is appreciated.

Codehaus' instance of Confluence does not allow public registration, so even though I'd like to weigh in on this discussion, I am unable to do so there. May I suggest we keep this in JIRA so that cobertura plugin users are able to contribute their suggestions as well?

Steven Swor
added a comment - 19/Jan/14 6:10 PM Codehaus' instance of Confluence does not allow public registration, so even though I'd like to weigh in on this discussion, I am unable to do so there. May I suggest we keep this in JIRA so that cobertura plugin users are able to contribute their suggestions as well?

The cobertura-lifecycle has to be extended, and I think it should look something like this:

Before entering the test phase, the java-classes need to be instrumented for test
Before entering the integration-test phase, java-classes need to be instrumented for integration-test.
Now they both have their own .ser file, and during verify all these should be checked and eventually reported.
I think this would result in less configuration and no need to misuse a report-only goal.

This sounds like a reasonable approach. What sort of work would be involved in implementing this solution?

Steven Swor
added a comment - 20/Jan/14 8:52 PM
The cobertura-lifecycle has to be extended, and I think it should look something like this:
Before entering the test phase, the java-classes need to be instrumented for test
Before entering the integration-test phase, java-classes need to be instrumented for integration-test.
Now they both have their own .ser file, and during verify all these should be checked and eventually reported.
I think this would result in less configuration and no need to misuse a report-only goal.
This sounds like a reasonable approach. What sort of work would be involved in implementing this solution?

Today I needed the ability to measure test coverage for integration tests. I have applied CoberturaIntegrationReportMojo.patch and cobertura-maven-plugin_check-only-and-report-only-mojos_with-IT.patch locally, and made some adjustments because the code in SVN has changed since the patches were made. After that I could get a very welcome coverage report for our integration tests.

I would like to apply the integration test coverage bits to trunk, however looking at the patches I have a few questions.

In CoberturaIntegrationReportMojo.patch there is a new Mojo that can be used to measure coverage for integration tests, like the subject is for this issue. Am I correct when I say that this is the only patch that I need to apply to solve this issue?

In cobertura-maven-plugin_check-only-and-report-only-mojos_with-IT.patch there are a couple of new Mojos CoberturaReportOnlyMojo and CoberturaCheckOnlyMojo complete with integration test. But I don't quite understand their purpose, in relation to integration tests. Are they in fact an entirely different issue? Also their Javadoc are exactly the same as the mojos they extend, which makes it hard to understand what they do.

Dennis Lundberg
added a comment - 06/Nov/14 2:18 AM - edited Hi,
Today I needed the ability to measure test coverage for integration tests. I have applied CoberturaIntegrationReportMojo.patch and cobertura-maven-plugin_check-only-and-report-only-mojos_with-IT.patch locally, and made some adjustments because the code in SVN has changed since the patches were made. After that I could get a very welcome coverage report for our integration tests.
I would like to apply the integration test coverage bits to trunk, however looking at the patches I have a few questions.
In CoberturaIntegrationReportMojo.patch there is a new Mojo that can be used to measure coverage for integration tests, like the subject is for this issue. Am I correct when I say that this is the only patch that I need to apply to solve this issue?
In cobertura-maven-plugin_check-only-and-report-only-mojos_with-IT.patch there are a couple of new Mojos CoberturaReportOnlyMojo and CoberturaCheckOnlyMojo complete with integration test. But I don't quite understand their purpose, in relation to integration tests. Are they in fact an entirely different issue? Also their Javadoc are exactly the same as the mojos they extend, which makes it hard to understand what they do.

Issue created in March 2008, patch you're commenting was added in 2009, and now is almost 2015....
I'm not sure if issue is relevant any more at all. I'm not using cobertura anymore, partially because it took so long to get attention to this issue; jacoco works for me.
So, AFAIAC you can ignore patch. Removed my vote.

Stevo Slavic
added a comment - 06/Nov/14 7:05 AM Issue created in March 2008, patch you're commenting was added in 2009, and now is almost 2015....
I'm not sure if issue is relevant any more at all. I'm not using cobertura anymore, partially because it took so long to get attention to this issue; jacoco works for me.
So, AFAIAC you can ignore patch. Removed my vote.

Dennis Lundberg
added a comment - 06/Nov/14 8:25 AM Digging some more into the code it looks like we will also need a CoberturaIntegrationTestCheckMojo that extends the CoberturaCheckMojo , but with a different execution phase.
The clean , dump-datafile and instrument goals does not appear to require any changes.

Dennis Lundberg
added a comment - 06/Nov/14 9:09 AM Two new goals were added in r20181:
cobertura-integration-test (based on the patch by Stevo Slavic)
check-integration-test
They extend the cobertura and check goals, but change the execution phase so that integration tests are also included in the Cobertura analysis.
I will push new SNAPSHOTs to the snapshot repo a little later, so that people can test it out.