Running TestNG-, JUnit3- and JUnit4-tests with Maven Surefire in one run

Fortunately, the team I’m currently working in emphasizes on a high degree of test coverage. First we started with JUnit4 but switched to TestNG after a while concerning design and performance purposes. Although we migrated most of the existing tests, we stuck in the situation that there were still JUnit4- and JUnit3-tests which couldn’t be converted to TestNG. Those tests depend on third-party-libraries which in turn are related to JUnit.

So how to execute TestNG, JUnit3- and JUnit4-tests on the same compilation run using Maven?

Imagine a project with just one TestNG-, one JUnit3- and one JUnit4-test. Simply adding the Surefire plugin to the pom.xml (and of course the JUnit4- and TestNG-dependencies) will make Surefire use TestNG as default. Thus a call of mvn clear package will lead to:

As you can see, Surefire triggers just two of the tests (TestNG and JUnit3). That’s because:

TestNG executes TestNG- and JUnit3-tests only

JUnit4 executes JUnit3- and JUnit4-tests only

See [1] for details.

Solution 1: Profiles

So the first solution for this issue bases on profiles. Beside the default Surefire-run with TestNG there is an additional profile (here named “JUnitRun”) in which the attribute “TestNGArtifactName” is set to “none”. As a result JUnit4 will be taken by Surefire when calling Maven with the profile “JUnitRun”.

Although this provides a solution it also has a big drawback: you need two compilations to get all tests executed (mvn clean install and mvn clean install -P JunitRun). In addition the JUnit3-tests are executed two times (once a run). This isn’t acceptable for a continuous integration system. So how to join two executions to one?

Solution 2: Two executions in one run

Maven can trigger more than just one execution of the same plugin [2]. This is utilized in the following solution: there’s a (default) execution for running TestNG and a second execution which TestNGArtifactName is set to “none” (again for calling JUnit4 instead of TestNG). So the pom.xml looks like this:

But the drawback remains that the JUnit3-tests are triggered two times. In a project with many JUnit3-tests and/or some long running ones this is an indefensible circumstance. Also reporting tools might get confused while calculating their results.

Solution 3: Two executions, one restricted

Improving the second attempt can be done by deciding whether the TestNG- or the JUnit4-tests should be treated in a special way. Assuming that there are just a few JUnit4- but lots of TestNG-tests, the final solution is to call every TestNG- and every JUnit3-test within the default execution (as it is already done in solution 2). But now the automatic execution of tests in the second run is rejected. Instead the second execution triggers the JUnit4-tests explicitly by using the “includes”-tag. Inside those tags it is considerable to either list up the JUnit4-tests one by one or follow a team-agreement like “every Junit4-Test ends with *Junit4Test”—as it is done in the following listing:

Hello, thank you for an article. It is quite helpful.
I have a different question though – How to make a configuration to run only JUnit or only TestNG tests when ‘mvn test’ is called (I guess with some extra parameters pass in commandline)?

About this blog

The intention of this blog is to give clues and solutions to problems I struggle with in my every day work. As not everything is worth to write about it and I also don't want to post stuff which could be found on
every corner of the internet this blog is updated continuously but not regularly.

If this blog helped you in some way or you want to share your point of view to
a given topic—feel free to do so. I appreciate every comment and try to answer
as fast as my spare time allows me to do so.