Colliding Objects

Coding, design, and broken feedback loops

Jasmine-Node's Test Failures Explained 4 years ago

jasmine-node is a great testing framework. Still, the asynchronous execution model of its underlying platform, node.js, has some implications on the way it behaves once a failure has occurred, which may surprise you if you are used to traditional xUnit frameworks. Understanding these differences will help you avoid several painful pitfall.

Asynchronous vs. Synchronous tests

In jasmine-node every test (that is: every callback function passed to the it() function) can optionally receive a done parameter.

If your test does not issue async. calls, your function should not define this parameter. jasmine-node will detect that your function has zero parameters and will treat it as a synchronous test: it will assume the test has finished once execution returns from the function you supplied.

On the other hand, if your test does spawn async. flows, you must make your test function take a single parameter, typically called done, which is essentially a callback function supplied by jasmine-node. You need to arrange your test code to call this callback function once all async flows triggered by the test are over.

In jasmine-node, assertion failures are not surfaced as thrown exception. Instead, the assertion code simply registerthe failure with the framework (so that it can be reported later), but other other than that, execution continues normally.

The code above places the expect expression inside a try...catch block and stores the exception that was thrown in the captured variable. We then assert that captured is null. As you can see below, the test fails due to 990099 != 3, and not due captured being non-null, which indicates that an expect failure does not induce a thrown exception:

In xUnit such a situation leads to the first assertion throwing an exception and the second assertion not being executed at all. In the jasmine-node, things are different. looking at the output we see two separate failure reports and the summary line indicates two assertions:

Exceptions and side flows

This is a nasty minefield. By 'side flow' I mean a block of code that is asynchronously executed outside of the test's main flow. Such blocks are very common in node.js (arguably, this is a key part of its value proposition).

We'll start with properly written test, and then examine the effect of diversions. Specifically, here's a test that opens a web-page and checks its content.

(This is just a toy example to illustrate a point. In a "real" test, you will probably bring up an HTTP server first, then access it via a localhost:port address).

As most IO in node.js is async, the zombie API does not let the test's main flow see the content. Instead, browser.visit() takes a callback function that will be invoked once the content is available, which can be well after the test's main flow has exited.

Inside that (async) callback block we first check the page's content via

expect(browser.text()).toContain('Home');

and then call done:

done()

For the remainder of this section let us assume you made a silly mistake: instead of toContain you typed includes.

Omitting the --captureExceptions flag from the jasmine-node command line

$ jasmine-node spec/explore.spec.js # DON'T DO THIS
$ echo $?
0

yields an empty output despite the fact that your code still suffers from the includes()-instead-of-toContain() bug. It is as if the test was never invoked. The exit code (as reported by echo $?) is 0 which indicates "all tests are passing". The problem in your code is no longer detected by your tests. Very bad.

Don't try this at home #2: Calling done() from the main flow (when a side-flow exists)

What happens if you move the done() call out of the side-flow and into the main flow?

This is the worst possible outcome: we get a Green line telling us everything went well and the exit code is zero. The only indication that something went wrong is the 0 assertions printout which is too easy to miss (and is never checked by automatic tools such as travis).

Bottom line

Always run with --captureExceptions.

If you start a side-flow make sure it calls done() (and that the main flow does not).

If there is no side flow (no async call) your test-function should not declare a done parameter.