equivalent functionality available using the
xmlrpc or rmi AR interface

Repeatability

repeatable

repeatable

dependent on remote services

depends

Test should be named

xxxUnitTest.java

xxxIntegrationTest.java

xxxSystemTest.java

xxxTransportTest.java

added to a suite named

AllxxxUnitTests.java

AllxxxIntegrationTests.java

AllxxxSystemTests.java

AllxxxTransportTests.java

Details

tests a class in isolation, outside the
hivemind / workbench framework. No external resources or network
services to be called. Other dependencies should be mocked or
stubbed. If the test needs to fetch a url, provide a classpath
resource and pass it the url of that.

exercises a class within the hivemind /
workbench framework -either via its public interface, or other
backdoors. No external resources or services to be called. Other
dependencies should be provided by hivemind system. Use
ARTestSetup.javaas a fixture

Test a class within the hivemind / workbench
framework using it's public AR api where possible. Use
ARTestSetup.javaas a fixture

Repeats a system or integration test but
drives it using the RMI or XMLRPC interface instead of the
direct-function-call interface. Implemented by subclassing an
existing system or integration test and changing the connection
method.

Test Organization

In the
org.astrogridpackage there are top-level test suites,
which provide a handy way to run a large amount of tests in one go.
These can be run from within Eclipse, or from the commandline using
maven.

AllRepeatableTests

The useful set of tests while developing. 100% pass, 5 minutes
to run, 30% coverage of the source (as of 26/6/07) .Comprises -

AbsolutelyAllUnitTests

A fast suite of all unit tests. Expect 100% pass

AbsolutelyAllIntegrationTests

A fairly fast suite of all integration tests - useful for
detecting hivemind errors. Expect 100% pass
Slightly twitchy - you may find a run occasionally
fails - usually because they require login to the community service
(this should be coded out).

BestSystemTests

The most useful / reliable / uptodate of the system tests.
Slow, and likely to fail in places at the moment.

The most important one to maintain is
AbsolutelyAllUnitTests, which forms the automated tests for the maven build. Maven
runs this test suite after compiling the source. I hope to use
AllRepeatableTestsas the maven build test suite, but need to rework to remove
the dependency on external services (community) first.

Occasionally, it's necessary to comment out a failing test
so that the build will proceed. I try to keep this to a minimum,
and mark it with a
@fixmeto come back to later.

Hand Running Tests

At present, the system & transport tests are liable to
fail at various places, depending on various services being
accessible., and take a very long time to run in their entirity.
This means at the moment these tests are only useful in development
- not as part of the build process.

UI testing

After some thought, I've decided not to write JUnit tests
for UI code. It's hard to do, and will only catch what the tests
are written to catch - not formatting problems, layout errors, etc,
etc. The only approach for the UI is to hand-test.

Because of this, it's important that the non-visual parts of
the UI (datastructures and controlling logic) should be factored
out into classes separate from the UI, so that they can be
conveniently unit tested.

Smoke Tests

After the maven build assembles the application jar (which
contains the desktop code plus all libraries), it uses a stripping
tool to remove unused classes and code, to reduce the download
size. At the moment, this reduces the size of the full
DesktopAstroGridfrom a whopping 35M to 20M. With finer tuning
of the stripping tool, this size could be reduced further.

After stripping, a subset of the repeatable tests are run by
maven as a
Smoke testto verify that no required classes have
been stripped. If the smoke test fails, the build halts.

Hand-testing Checklist.

Assuming the smoke tests succeed, the build goes on to
produce an installer jar. I then go through a checklist of testing

plastic - try running topcat - see if it connects to the hub.
likewise for aladin.

check whether a notification shows up in system tray.

web interface. Open in browser,

navigate around,

exercise preferences,

try calling a function - e.g.
Webserver > getURLRoot

click on the
?button, then on one of the items in main UI

verify the browser displays correct help.

quit the application.

verify that aladin notifies of closure.

TODO: collect some test scripts that exercise a fair portion of
AR

astroscope.

enter position:
crab, radius
0.01* expect position to be resolved to
coordinate

run search

verify that buttons for aladin and topcat appear in LHS.

check history

switch to hyperbolic view (as it's still downloading).

switch between degrees and sexagesimal, and observe display and
inputs change

switch to services tab. select a service and click the link -
verify registry viewer appears for this entry.

verify that there's not overmany errors in the services -
should expect < 10%

click on the 'tasks indicator' to see list of pending
queries.

back to hyperbolic view. try moving through the tree, and using
the 'go to top' button

Platform Testing

For each variant of workbench, repeat a subset of the manual
testing checklist on each target operating system - Windows XP, OS
X, Ubuntu Linux. For XP and Ubuntu, verify that system tray icon
appears.