Implemented features

There are pre-integration-test and post-integration-test phases for preparing and cleaning up integration tests.

Proposed features to implement (short term)

A project should be allowed to have both unit tests and integration tests sitting next to each other.

Directory structure:

There's no need for introducing a third type of tests. Functional tests should be considered integration tests. If there's a need for more types of tests, it should be implemented by adding features to the surefire plugin to configure it in different executions but the 2 phases to attach tests to will be test (for unit tests) and integration-test.

Modify the surefire plugin to add a new surefire:it goal. See MSUREFIRE-50.

Add new <integrationTestSourceDirectory> to the POM model. See MNG-2009. Note: Does this need updating the minor version of the POM version?

Open short-term questions

Should it be src/it, src/itest, src/test-integration or src/integration-test?

Should the surefire plugin goal be surefire:it, surefire:itest, surefire:integration, surefire:integration-test?

Should we add a integration-test-package phase too?

Do we add a new it scope (or itest or integration-test)? (see MNG-2011)

Ideas for the future

An alternative directory structure:

Should we rename src/test into something more explicit like src/test-unit? BTW the same question will arise for the lifecycle which is currently named test even though it's meant to be only for unit tests. Same for <testSourceDirectory>.

Misc

Other issues/articles related to the testing strategy / testing best practices

I consider that Cactus test on my ejb are unit test. Some, because done inside a container will consider it integration test even if a test affect only one ejb at a time. I think the line between unit/functional/integration is sometimes to thin to base the directory structure on it. Why not have a test directory with inside it a separation by type of test such as:
test
|_junit
|_cactus
|_httpUnit
|_JMeter
|_...
and so on. The surefire plugin could then call the plugin responsible for cactus, httpUnit....test. It makes the structure clear and simple and the possibility to add test easy.

I really like Rahul suggestion since the division of test can be so different from an one organisation to another. Some might want to separate their tests by frameworks, some might still divide their tests by type but want a way to decompose their junit in two phases of test, ... I think Maven should give some flexibility here and let the user decides. Maven should just has some predefined lifecycle phases for each kind of test and a set of test could be bind to a specific phase by adding a xml element like the one used to bind plugins to a phase.

In my mind, the main information that needs to be captured here is the project knowledge regarding acceptable test suite success rate for a successful build.

Since unit tests verify the semantics of code, much like the compiler verifies the syntax of code, all unit tests must pass at 100%, just as they must all compile. Integration tests, by definition, test more than one part of the system, and are not required to pass at 100% for the build to be considered successful. Most likely a release would not be performed until all the integration tests pass at 100%, in fact they might be coded early in the development cycle and then treated as exit criteria for the release.

Then, I would use regular Java packaging to separate the test code for each different technology, such as JMeter, HttpUnit, and so on, if I felt that distinction was important. Perhaps I might prefer to split these by purpose instead, such as performance, web, and so on. The test technology separation doesn't seem to be a decision that Maven needs to be aware of, or something that Maven should try to enforce / standardize.

The test technology separation doesn't seem to be a decision that Maven needs to be aware of, or something that Maven should try to enforce / standardize.

From Alexandre Poitras' comment above it makes sense that:

Maven should just has some predefined lifecycle phases for each kind of test and a set of test could be bind to a specific phase by adding a xml element like the one used to bind plugins to a phase.

Testing Plugins (JUnit/Surefire/JMeter/HttpUnit) could be bound a test -> type (my comment above) with allowance for overrides if required. I believe this would allow Maven to treat any type of 'test' as a 'test' (no special handling) and delegate handling based on its type to the bound plugin(s).

Yeah but my concern is that a tool can be used for more then one type of test. For instance, junit is often used of course for unit testing but also for intregation testing (this is a common case if you use Spring framework). That's why I was proposing something similar to the plugin model but we could use your method for default values, easily overridable in the pom. This way the testing would be consistent with the pom global approach. The test type element would be like the project packaging element, it inferes default value about your tests but those values can be easily overrided by declaring some plugins and binding them to an execution phase. I think it would be a great solution not changing drasticly the pom format.

Here's a little example of what I mean :
*By the way, forget my comments about having a tool used for two types of test. I taught you were infering the test type from the tool, but I see now you actually meant test type-> tool. Anyway, what I just proposed still stands.
<build>
<sources>
<source>
<directory/>
<output/>
</source>
</sources>
<tests>
<test>
<type>Unit/IT/Functional/System/Custom</type>
<directory/>
<output/>
</test>
</tests>
<plugins>

I just thought I'd add to these comments the approach I'm currently using to perform integration tests with M2 as M2 is now. It may ensure that all angles are covered in any of your recommended approaches.

I have achieved integration testing (using both JunitEE/Cactus and JUnits calling remote interfaces) in M2 using a very simple lightweight POM for each discrete task/execution in the integration phase (i.e. bound to this phase) e.g.

All plugins defined under 'runtests-integration' (and below) are bound to the integration-test phase

I found this approach also necessary, because you can only have one instance of the antRun plugin for the integration-test phase, therefore you can't use antrun, then surefire, then antrun again as the last antrun overrides the first if they are bound to the same phase.

I did want to use Cargo, but unfortunately there was no support for OC4J with an existing configuration.

I have the above 'runtests-integration' suite only execute if the M2 build profile is set to -Pintegrationtests. In this way developers can run a regular M2 build which just runs standalone JUnits, or specify the profile to run a fuller build. We have CruiseControl always running the integrations tests profile.

With m2.02 I should now convert the Antruns to Ant Plugins.

I'm hoping that when M2 supports integration tests better, I can migrate to the approved strategy, I know this approach is a slight misuse of M2, but this works right now, we have 2 large Applications (both EARs) that are built with M2 via CruiseControl, the above integration projects start the container run all the integration tests (with the 2nd EAR application even invoking the 1st EAR's services)

I have another M2 module in addition to those shown above that actually hold the integration tests :-

I've been looking for a way to get my complex integration tests up and running and the only way this was possible was to use the strategy employed by Pete: Have a subproject for each step in your actual integration test lifecylce.

Right now it's impossible to get your tests up and running in a single module due to the well known bugs. But even if all bugs were fixed it doesn't make a lot of sense with the maven 2 model for the simple reason that there are not enough lifecycle phases and they have the wrong names.

Complex integration tests require dozens of steps to set up and tear down the testing environment.

I would propose to make the lifecycle configurable or the ability to just switch it off and run a sequence of plugins like me good old ant file.

But if the integration test directory is ismply called "integ" it will appear before the main directory in most IDE. The chosen name must be such as its comes after "test" in alphabetical order. Thanks!