Tuesday, 22 December 2015

Acceptance Test Is Not an Environment

In a traditional software development process where
you did analysis, development and then testing, there is often the use
of shared environments, and therefore there is often a one-to-one relationship
between the name of the environment and the type of testing performed. For
example UAT (User Acceptance Testing) tends to come right at the very end of the
process just before production. If you are working on a back-end system there
may well be no “U” in the UAT and so it really just becomes a more
production-like test environment.In a modern development process there is more of a
distinction between the type of tests we are running and the environment in
which we are running them. We are always trying to achieve a balance between
getting the fastest feedback possible on whether our changes are correct, whilst
still ensuring that enough of the system is being tested in a manner similar to
production so that we minimise any problems due to environmental
differences.In my C Vu article “The
Developer’s Sandbox” I described a number of different ways that you might
partition a system (and test data) to allow a variety of different levels of
non-unit testing. In essence I am mostly interested in running fast, automated
test suites in some isolated manner to gain rapid feedback. However I also like
to do a bit of manual exploratory testing, especially when making changes around
deployment or infrastructure code. And demoing new features is also important
too to ensure that we’re building “the right thing”.What I’ve found is that there is often some
confusion when talking about testing that conflates the suite of tests being
exercised with the configuration of the system it’s being run on. For example I
will try and run every automated test possible on my local machine before
committing my changes. This means I’m probably running some combination of unit,
component, integration, acceptance and system tests against a variety of mock
and real components and services depending on how expensive or not they are to
use.Similarly on the build server we will run exactly
the same suite of tests but because we have more time we can use the real
dependencies where possible and only rely on mocks where we have to. The closer
the code gets to production the closer the test environment has to get to
production too.As a consequence this means there is no one-to-one
relationship between the test suite configuration and the environment where it
is run. By default we tend to optimise for the developer feedback loop which
means the out-of-the-box configuration is usually “localhost” everywhere [1]. In
contrast the build server, development and test environments will likely have
real networks, databases, message queues, etc. in play and so the same suite of
tests will increase the amount of infrastructure and integration for a more
production-like quality, perhaps at the expense of performance. The point is
that we aim to run the same tests and only vary the configuration.
Hence when talking about automated testing it may require us to qualify it with
the environment configuration we might be running with to avoid
confusion.One natural observation might be that it’s not
right to call the running of the acceptance test suite on a developer’s local
machine “acceptance tests” as some element of the “acceptance” must come from it
being run in a more-production like manner. Whilst I get the sentiment, I think
that misses the point about developer’s leveraging the traditionally more costly
tests in a constrained, but by no means useless environment, to gain earlier
feedback around the functional behaviour. No, it doesn’t mean it’s
signed-off and ready for production just because it works on my machine, but it
does mean that at a fundamental level the change is sound and worthy of pushing
further down the deployment pipeline.

[1] I always say that I should be able to unplug
from the network and go out into the garden where there is no Wi-Fi and still be
able to write code and have a high degree of confidence that it works. Modern
tooling (and a sane approach to developer licensing) makes that possible even
when databases, message queues, etc. are in the equation without having to
restrict ourselves to relying solely on unit testing.