System Tests

System tests exercise a complete software application, verifying external interfaces like Web pages, Web service end points, or GUIs, work end-to-end as designed. Because these tests exercise an entire system, they are often created towards the latter cycles of development. These tests also have the tendency to have prolonged execution times, in addition to extended set up and configuration times.

For example, the following test utilizes
jWebUnit to test the login functionality of a Web site. Note that a number of assumptions are made in this test, such as the URL being available and the user "tst" actually having a valid account, not to mention a trading history. These implicit dependencies usually require a configuration step preceding a test run.

Developers should run these tests locally on an as-needed basis and in a continuous integration environment. Executing these tests nightly (if they can be pulled off in an automated fashion) is a good strategy. Running them more frequently can tax system resources, especially in larger environments. With proper hardware planning and sophisticated automation, however, teams can run these tests more frequently.

Implementing TestNG Categorization

Categorizing TestNG tests into these three categories is as easy as using the
group annotation described above. Having a test class with methods spanning various test granularities is unusual, therefore tagging can be effectively applied at the class level.

For example, the following class has been tagged as belonging to the
unit test group. Note how
HierarchyBuilderTest verifies the behavior of the
Hierarchy class by depending on the
HierarchyBuilder type. As this relationship ends at
HierarchyBuilder, which doesn't depend on a file system or database, you can effectively consider this a unit test:

Running categorized tests

Before checking in code to a content management system, it's paramount to run tests locally, either through a build or through an environment such as an IDE. Running categorized tests via the TestNG Eclipse plug-in is amazingly simple. By selecting the groups option in the TestNG Create, manage, and run configurations dialog, as shown in Figure 1, the list of available groups is presented with checkboxes to facilitate selecting one or more. After the desired group or groups have been selected, hit the Run button, and watch that green bar go!

Running categorized TestNG tests via a build becomes a matter of defining the appropriate
Ant targets for each group. For example, to run all tests belonging to the component group, the TestNG Ant task is defined with the
component group specified:

Accordingly, using this strategy, at least four test targets would be created. There would be three corresponding to unit, component, and system tests, and a final target that could run all three.

That's a Wrap

TestNG facilitates test categories quite easily and this is probably one of TestNG's most exciting benefits. What's more, TestNG's
group annotation also helps place tests into other categories, such as batch tests, acceptance tests, and even performance tests. In fact, it seems this feature may have had an influence on the newest version of JUnit, which is planning to support test groups as well!

The next time you blindly code a new test case, consider the long-term implications on build execution times. Build scalability starts with a test categorization strategy that is run at various frequencies, and an effective test categorization strategy starts with TestNG.