Understanding JUnit method order execution (updated for version 4.11)

Let’s start with a rule: You should not create test methods that depend on the order in which they are executed.

If you do have such tests and they are failing randomly or rarely, this is why: JUnit relies on Java’s reflection API to get which test methods to execute. The problem is that the API does not define the order of the methods it returns. Your tests may work for a long time and then fail, apparently randomly. The things is, you’ve just been lucky all along and relying on the Java run-time giving you a consistent answer when it makes no such guarantee.

You may even see some very confusing behavior like a superclass’ test methods being mixed in with its subclass. I’ve seen this in Eclipse and Ant for example.

JUnit 4.11 provides a workarounds for this behavior with a new class-level annotation: FixMethodOrder(MethodSorters.NAME_ASCENDING). For example:

Post navigation

7 thoughts on “Understanding JUnit method order execution (updated for version 4.11)”

Now JUnit has added an annotation which we can use to have the tests run in alphabetical order. Well, that’s an idea. But that’s not what I want. I want to have the tests run in their order of appearance in the class. Is there a way to do that ?

The isolation of tests is a self-correctiong mechanism to maintainable source code. If you have a problem with isolation of tests you may have isolation problems in your production code and it will be less maintanable.

On the other hand you will achieve isolation in your production code. Developers have to decouple things to make them isolated testable.

Isolation supports expressive tests that shows the problems at the point the test adresses. If you have dependent tests you may not find the cause for the problem at the point where the assertion fails.

That are really big issues and they will be larger the more source code you have.

So keep your tests independent from each other! And I would not rely on the presence of @FixMethodOrder in next major releases of JUnit.

There is more than meets the eye on issues like this.
In one case, you may be (as I at work) relying on third party libraries that contain bugs, various side effects, and oddities that arise when your code use their APIs. In another, you may be working in a larger team and another team’s code may contain side effects but the fact that they show up in your test case may not be a prime concern for them because your use case is not important. Maybe it will be later, but not today.
Regardless of how you get here, there is also the cost of debugging and rewriting tests, or worse, debugging and rewriting production code to handle executing test methods in any order. In any project, you need to prioritize issues like this against new features, bug fixes and general maintenance.
So yes, if I were to write new code with no dependencies, I would account for executing unit tests without the fixed order directory, of course! Even then, that would not guarantee that the JVM would run my tests in random order and that they would pass. Even without fixed order execution, a given JVM executes the test methods in the same order until you change the some part code.