CFML, Clojure, Software Design, Frameworks and more...

TestBox - replacing MXUnit and providing BDD

December 23, 2013 ·

If you've seen my tweets over the last couple of days, you'll know that I've been taking a look at TestBox, the new testing framework from the ColdBox team.

You might be thinking "Do we really need a new testing framework?" and it's a reasonable question. CFML has had a number of testing frameworks over the years but mostly we've seen each one come and go as maintainers have moved onto other things: cfUnit, then cfcUnit, then a brief resurrection of cfUnit, then MXUnit. Does anyone remember cfSpec? I liked that a lot - I like the Behavior-Driven Development style of writing tests. That's why, at World Singles, we use the Expectations library for almost all of our Clojure-based tests.

So why the interest in TestBox? My initial interest was spurred by the inclusion of the BDD style of tests which I find both more natural to read and a better fit for expressing requirements for software that is yet to be written. Then I read that TestBox has MXUnit compatibility which intrigued me - we have been using MXUnit for several years at World Singles and I don't relish having to rewrite all those tests. If TestBox could allow us to run our existing "xUnit" tests as-is while we write new tests in the BDD style, I'd be very happy.

Adam Cameron has been blogging about Test-Driven Development recently and has also picked up on TestBox. He just blogged about trying TestBox on his extensive MXUnit test suite and you can see he ran into a number of issues, all of which he has reported, and which I expect we'll see fixes for. My experience was more fruitful than his - as I mentioned in my comment on his blog post: just two small changes got the whole of our suite running flawlessly!

My next challenge was integrating TestBox into our automated build / test script, which uses Ant. TestBox provides a runner-template that shows how to do basic Ant integration. I followed that recipe but discovered there was no way for Ant to see that any tests failed and so I couldn't "fail the build" if TestBox found errors or failures. After studying the source code briefly, I realized I could extend the TestBox CFC and provide a way for the runner CFM template to access the details of the TestResult object, and write a property file that Ant could read and fail if a certain property was set. Time to fork the ColdBox Platform Github repo and send a Pull Request to add build failing functionality which Luis merged in within a few hours! Nice!

With that in place, I was able to delete MXUnit from the World Singles repo and rely entirely on TestBox for all our testing needs!

Whether you prefer the "xUnit" style of tests, with assertions, or the "BDD" style with expectations, TestBox can support you, and even if you have a lot invested in MXUnit, migration to TestBox should be straightforward (especially once they address the issues Adam has raised). I think we can be sure of ongoing support and maintenance - based on the well-established track record of the ColdBox Team. I think TestBox is a safe bet. Now you just need to start actually doing some automated testing :)

4 responses

Interesting that you integrate the test runner with the build tool you made. Perhaps you develop without "building" for a long while, then you "build" when you are going to try to deploy to production.

Have you taken the time to test javascript the same way? Any preference on JS test framework? Most of the outstanding bugs I have are JS and browser specific.

In addition our build script can create a deployment snapshot artifact, and it can also deploy a snapshot artifact, and several other useful tasks. Our build script also runs Grunt to minify and concatenate CSS/JS etc.

As for JS, we don't (yet) have the sort of front end component-based code that would suit unit testing but I expect we will at some point. Although we'd most likely use ClojureScript instead of JS and use the testing framework that comes with that. Our WebDriver tests can be run against multiple browsers (although use an old Firefox version as a baseline for general testing).

That sounds impressive. The use of a build tool to minify/concat JS/CSS before it's pushed to the server - great idea. I currently do mine on server by tracking file changes instead, but I think you helped me realize it would be smarter to do it locally from the moment of build/deploy instead. I want to get my code compatible with clojure compiler so it can use the sourcemap feature - currently using yui, though most sites are not minified yet. I also have a css sprite map feature I built myself that parses CSS, and re-writes it carefully. Being able to reproduce a previous version of the entire environment in an automated way, that sounds awesome. If it was open source, a new community developer could setup the app very easy. I imagine that would be useful if you were responsible for multiple installations of the software or a larger team.

That's interesting that you have considered JS so far to be less important to organize&test compared to the backend. With all the jquery plugins now that get the job done quick, it's very easy for JS part of the code to become a mess - I think i have around 30-40k lines of third party JS in my app. Fortunately a lot of that third party stuff was already tested by others, and it's pretty reliable. If you use clojurescript, hopefully there is a source maps feature that will help with debugging, but if you have trouble with the code it generates in an older IE browser that doesn't have source map support, you would be more frustrated compared to debugging your own javascript probably. This is why I'm scared of Dart & ClojureScript, etc.

I just started thinking of better code quality in javascript and I'm using netbeans for editing JS now - it's very nice. I just learned about window.onerror recently, and I started logging all of those, and fixing them. That's helping a lot. I don't know if your webdriver is setup to automatically log any js error or if it just validates specific elements/page states, etc.

Other js language spin-offs, may not having the same level of tooling as javascript because they are so new, and you become more tied to 1 or 2 vendors again, making it less open. Dart is necessarily going to be useful for general web development for many years and it's only purpose it to be like twice as fast really. I feel like just writing better javascript is the right answer for now for me. I learned a lot investigating how jquery & its plugins were written in the past. Javascript is usually more painful to work the bugs out, since the testing effort is multiplied. I used to like flash because you get the same result everywhere, but unfortunately, there is nothing to make javascript easier other then selenium I think.

I expect more of my future work to be javascript oriented since the best apps are a continuous experience instead of a page flipping. I just easily run out of time making the cool JS stuff. It's easier to make money doing things as simple pages.

The concept of using a build tool to concat / minify stuff is certainly not new - if anything, I feel we've been a bit late to that party :)

When you talk about code compatibility, I assume you mean the Google Closure compiler ('s' not 'j', not to be confused with Clojure and ClojureScript).

I did not say I thought JS was less important to organize and test - I just said we don't (yet) have much JS code: we use a handful of libraries / plugins but there's very little of our JS code. We're unlikely to write a lot of JS - for the reasons given above, we're more likely to write ClojureScript, and rely on its testing frameworks (as well as its superior approach to code organization and semantics).