JUnit Is Not Just For Unit Testing Anymore!

Non-Java tools are often used for functional (black box) regression testing J2EE web applications. Such tools include Mercury WinRunner, SilkTest, (IBM) Rational Robot, and others. The lure of these tools is the “capture/playback” capabilities (making it easy to create tests without writing code), and scripting capabilities.

This seems like a positive step away from tedious, error-prone, and infrequently done manual regression testing. But you may be able to do better.

I label these tests semi-automated (or semi-manual if you’re a “half-empty” type) because of the way I see them applied in practice: I see them initiated manually, followed by interpretation of the results by the team’s “QA person.”

These tools have some disadvantages:

I’ve yet to see them plugged them into a Continuous Integration server such as CruiseControl or Continuum. It is reckless and wasteful not to run regression tests continuously (think hourly) while developers are changing code, especially across multiple teams. The Continuous Integration server needs a way to create a known initial state (i.e. build, deploy, and reset the database), initiate the tests, roll up all their results (along with results of unit tests) into one binary pass/fail assessment, and identify which test has failed when that occurs. These things are relatively easy to do with JUnit-based tests. It should also be technically possible to do this with WinRunner/SilkTest/etc. but I haven’t seen anyone do it yet. If you’ve gotten this to work, please write to me or post a response here.

License restrictions impede the reproducibility of the test results. In XP teams everyone has access to their own “push to test” button, and pushes it a lot.

License restrictions and proprietary scripting languages re-enforce the unagile idea that one person on the team is responsible for testing. It’s hard enough to break this habit without artificial restrictions from the tools.

Overuse of capture/playback can make the tests break easily when user interfaces change. This can be mitigated by using the proprietary scripting languages.

JUnit tests are easy to plug into Continuous Integration, aren’t bound by license restrictions, and are written in Java, one of the best known programming languages (especially on teams that are already developing Java applications!). Most traditional testers have no background in Java, thus would need to pair with Java programmers to create tests.

The JUnit framework was originally intended (and still primarily used) for “unit testing” — that is, white box testing of individual components outside the context of the whole system. That’s great and all, but creating deterministic (very thoroughly tested) control systems for aircraft and spacecraft has taught me that systems tend to fail more than individual components. We need end-to-end (system) tests even more than we need “unit” tests. This is one area I differ (slightly) with the mainstream XP crowd.

The good news is that a great selection of third-party libraries exist for system testing with JUnit. They’re all free and/or open source.

JUnit alone is sufficient for EJB testing. As with the others, your build script will need to deploy and start the server. And you’ll need some way to reset the database to a known state. Apache Cactus may be useful in some cases.

Watij will drive a real Internet Explorer browser in the JUnit framework (through Java code). This is pretty cool because so many bugs are related to browser behavior, especially JavaScript. Watij is still kind of new and lacked extensive support last time I used this. (We’ve also tried a Ruby-based tool if you’re into that kind of thing.)

Selenium will drive a wide array of browsers for MS-Windows, Linux, and Mac OS X. This, combined with Selenium Remote Control is the current favorite at Danube.

actiWATE will emulate a web browser, including JavaScript (AJAX, etc.), in the JUnit framework. This might be useful when the tests need to run headless. actiWATE is free but not open source like the others. I found actiWATE easy to use.

Abbot is for testing Java Swing and AWT GUIs, such as desktop applications, Java WebStart applications, or Applets. The ScrumWorks desktop client is extensively tested with Abbot on MS-Windows and Mac OS X.

When someone at Danube checks in a code change, the screens of a Mac and a PC in the back room (each running CruiseControl) wake up and start GUI tests doing many of the same things you do as a user, except faster and much more abusively. And yeah, we still need to do a lot of manual testing.

Those are the third-party tools/libraries we can vouch for from personal experience. If you’ve had good experiences with others (or bad experiences with any of these), please let me know!

Michael James is a software process mentor, team coach, and Scrum Trainer with a focus on the engineering practices (TDD, refactoring, continuous integration, pair programming) that allow Agile project management practices. He is also a software developer (a recovering "software architect" who still loves good design).

8 comments on “JUnit Is Not Just For Unit Testing Anymore!”

i am a qa person and i am totally sold on junit and the benefits of continuous integration. however, you omit a key benefit of tools like silktest, winruuner, etc., which is that they support data-driven tests. this for qa is imperative. we need to test many permutations of an application event, not simply a few positive ones. for example, when working on an ecommerce site and testing a search engine, i want to script the invokation of the search function just once and make the input string a parameter, where that parameter is fed by an external data file that can contain a large number of test strings. we need to test way more than one or a few positive test cases, and data-driven testing allows us to do this. you have mentioned to me, michael, that we should look into FIT for this purpose, and i will, i just want to call out that for qa people this is a critical piece of the puzzle.

Carol’s message illustrates the value of involving QA from the very beginning. An isolated programmer in a hurry to get “code complete” is less likely to think of the negative test cases necessary to get work done/done/done.

While many JUnit teaching examples focus on the positive cases (what we used to call the “happy path” or “main success scenario” back in the days of use cases) it’s just as applicable for testing error handling.

I applaud that professor for teaching automated testing (which should be an integrated part of programming) in his programming course. I’d be a bit wary of continuing the System.out.println() approach beyond one course though. JUnit’s not too hard to learn, especially compared to learning to code.

Just a quick comment about data driven tests with watij or other tools.

Since watij is a java api you are free to construct any type of testing framework. You can model your automation framework to mirror your applications model and build test cases which are also data driven. It would not be difficult to build a test case which pulls data from a file or internal data structure to execute a test multiple times with varying data scenarios.

For a simple example: I could model a Login with a LoginTest class which uses a User object to login and validate on login. You could then create multiple scenarios for LoginTest by just creating new User objects with varying scenarios and validate expectations.

Your point illustrates there’s a lot to be said for writing the tests in a programming language like Java, which lets you do *anything*, rather than a proprietary scripting language which seems easier to learn at first but is ultimately more limiting.

Congrats on the 3.1.0 release of a great tool, and give a great talk at Agile 2007.

nice topic, testing is always a major hurdle to get ‘leet’ programmers to become business useful.

http://www.junitee.org/ seems missing from your list though. Cactus does the same, but the core JunitEE code is nice if you want to integrate the test code right into the program environment and get away from unit testing. (free too)