Be careful when using JUnit's expected exceptions

Sep 26th, 2012

For many people, JUnit is the grand-daddy of testing frameworks. Even
though other testing frameworks came first, a lot of people got their
start with JUnit.

People often start out testing with simple Boolean assertions, then
move on substring matching, then maybe on to mocks and stubs. At some
point, however, most people want to assert that their code throws a
particular exception, and that’s where our story starts.

When JUnit 3 was the latest and greatest, you were supposed to catch
the exception yourself and assert if no such exception was
thrown. Here’s an example I tweaked from Lasse’s blog and the
JUnit documentation for @Test.

Both of these forms offer a lot in the way of conciseness and
readability, and I prefer to use them when I need to test this kind of
thing. However, both forms can cause a test to pass when it shouldn’t
when the code can throw the exception in multiple ways:

123456

@Test(expected=NullPointerException.class)publicvoidtest_for_npe_but_which_one(){CoolObjectobj=newCoolObject(null);obj.doSomeSetupWork(42);// What actually throws the exceptionobj.calculateTheAnswer();// What we want to throw the exception}

In languages that have lambdas or equivalents, this problem is easily
avoided. For example, you can use expect and raise_error in RSpec:

Alternate solutions

Until a version of Java is released with lambdas, I see no better
solution than using try-catch blocks, the old JUnit 3 way. You could
define an interface and then create anonymous classes in the test to
have the desired level of granularity. This is a pretty bulky syntax,
any variables you use in the object would need to be declared final,
and then you have to explictly run the code!

If you can rephrase the problem slightly, you might be able to use the
fact that ExpectedException can assert on the exception message to
restrict your test. If you know that only your error can include a
certain string, then checking for that string could prevent tests from
passing when they shouldn’t.

Another solution would be to modify your code or tests so that you
don’t have to deal with the problem in the first place. If you can
move the setup code into a @Before block, then the exception
wouldn’t be caught by the test. If you can change your code so it
cannot throw the exception multiple ways, or if it throws different
exceptions, then that would also allow you to sidestep the problem.

Update 2012-09-27

David Bradley points out that if you configure the JUnit rule right
before the expected exception, you can reduce the possibility of
error. Unfortunately, exceptions thrown after the desired line will
still cause the test to pass incorrectly. This may not be a problem in
practice, as you are unlikely to continue the test after an exception
should be thrown, and most Java tests do not have a teardown phase.

12345678910111213

@RulepublicExpectedExceptionthrown=ExpectedException.none();@Testpublicvoidtest_for_npe_with_rule_at_last_moment(){CoolObjectobj=newCoolObject(null);obj.doSomeSetupWork(42);thrown.expect(NullPointerException.class);obj.calculateTheAnswer();// Any exceptions here will still cause the test to pass incorrectly}