Marty Andrews

artful code

Tuesday, March 29, 2005

Expectations on test speed

Unit testing

Hundreds - thousands of tests should be able to run in the order of seconds to get very fast feedback on changes.

The speed is facilitated by using mocks to provide expected behaviour from other services nearby.

The use of mocks is facilitated by dependency injection techniques.

A side effect of this technique is that design has very low coupling. A code base that achieves this is likely to have many interfaces. Developers are often confident to refactor code or otherwise make changes quickly given that they have a large safety net. They will usually run unit tests many times per day on their development machines.

Integration Testing

Test groups of classes or components working together.

Hundreds of tests should be able to run in the order of minutes

These tests are usually at least run as part of the continuous build. Sometimes developers will also run them on their development machines based on their judgment about whether their work may impact groups of components working together.

System Testing

Test the system as a whole.

Tens of tests should run in tens of minutes or hours.

These tests are run on a regular basis. Ideally this will be part of the continuous build, but sometimes this becomes difficult (tests might take too long, or full environments may not be available all the time). If this is the case, they may be run overnight, or on weekends. System tests can take the form of HttpUnit or JWebUnit tests for web applications, or testing technologies that interface with other protocols. Applications with thorough system testing should experience low regression figures.