Functional Testing in a Containerized World

Aug 18 2015

In an age of REST, IoT, and cloud services, well constructed functional tests
can exercise substantial portions of the code by simply emulating a client of
the service, and then testing the environment. But the real challenge of
building good functional tests is creating a clean and reliable test
environment that reflects the actual deployment platform. And this is where a
couple of popular cloud technologies can provide a huge boon to functional
testing. With virtualization (e.g. EC2) and containerization (Docker),
developers can create disposable and repeatable testing environments that
reflect production.

Software developers have a broad taxonomy of testing. Unit tests exercise
individual software components in isolation. Regression tests provide a
safeguard against the re-appearance of known issues. And functional tests
ensure that an application as a whole is properly functioning. Well written
functional tests exercise the application fully, and for that reason they may
very well offer the most bang for the buck.

Developers typically focus their effort, however, on unit testing. Perhaps this
is because they provide a fine level of granularity, and are often the easiest
form of testing to code, unit tests are justly popular. Thanks to movements
like TDD (Test Driven Development), most developers feel obligated to write at
least some unit tests. And, in fact, medium and large projects commonly run
unit tests automatically each time code is checked into version control. Unit
tests are good, but they do not (by design) guarantee that the whole
application is functioning as intended.

Functional tests, in contrast, enjoy lower adoption rates. This is probably
related to the traditional workload ascribed to functional testing. Not only
must the developer write tests. There is also an environment to set up. This
comes complete with databases, message queues, caches, file systems, and
whatever other resources the application relies upon in production. And these
environments must be maintained while also being protected from the
side-effects of catastrophically failing tests. Because they take so much work
to maintain, typically only teams with dedicated QA teams bother with full
functional tests.

With the maturation of cloud services, though, a batch of alternatives are now
surfacing. The process of setting up and tearing down environments used to be
labor intensive, time consuming, and error prone. But cloud services make this
process scriptable and fast. Environments are no longer sacred ground to be set
up and maintained for long periods of time. They are disposable.

When it comes to running servers in the cloud, there are two major approaches.
The first is virtualization. This model is popularized by Amazon’s EC2 and
Microsoft Azure. Hefty servers provide a virtualization service which allows
entire operating systems to execute inside of this host. In this model, the
host system emulates the properties of a hardware platform so that the hosted
system believes itself to have its own network card, storage, memory, and so
on.

Another popular model has emerged, though. Containerization is similar in many
respects to virtualization, but with a notable difference. Instead of running
an entire operating system for each container, the host system runs a single
kernel, but each hosted system gets its own file system and process control.
Because more is shared and less is emulated, containers can be created and torn
down rapidly. Docker, CoreOS, HashiCorp, and Century Link Labs have each dived
into the containerization world and begun creating sophisticated tooling around
containers.

Whether virtualized or containerized, the promise of these tools is that we can
now think of operating systems as ephemeral. We can set them up and tear them
down as it is convenient for us. And thanks to a wealth of new tools, we can
automate this process with meticulous precision.

This is where functional testing becomes exciting. By crafting repeatable
builds for our virtual machines or containers, we can bring up a new testing
environment programmatically, then execute a full batch of tests, analyze the
results, and discard the entire environment. Each time this process is
initiated, the environment is pristine. And when a test fails catastrophically
and, say, corrupts a database, this won’t impact subsequent test runs. The
entire environment is destroyed and later re-created afresh.

The most exciting project illustrating this new approach to functional testing
is Drone. With commercial services at drone.io, and the entire code made
available as open source, Drone is paving the way to a new breed of testing
tools. Poised against the common but clunky Jenkins CI server, Drone uses
containerization to create entire environments. Using Docker build files and
images, developers can run multiple containers, each performing its part, and
create (not simulate, not emulate, but actually create) an entire network of
services.

For example, a REST-based web service may have a relational database, a web
server, file services, and a message queue. In production, each of these may
run on separate intastances. In Drone, a developer creates a main container
(likely the one running the web server) along with a number of dependent
services, like the database and the message queue. Scripted tests can then test
the database migrations, the initialization logic, and the functioning of the
service. All of this is automated. And once the tests are complete, the results
are reported back to Drone itself, which dutifully tracks test runs. Then the
test environment is destroyed.

Because these environments can be set up in minutes or even seconds, testing
does not become a lengthy ordeal. Developers can expect feedback soon after
each code check-in. And because tools like Drone are easily integrated with
version control systems like Git, functional testing becomes an unobtrusive
part of the development lifecycle. Developers simply push their code into the
source code control and the tests are automatically run.

What makes this approach so exciting? There are several reasons. Running tests
in isolation, and in a controlled environment, is highly desirable. Because the
environment itself is scripted, the testing team gains programatic control over
it, too. An environment like Drone allows developers and QA to focus on writing
good functional tests, rather than maintaining an extra set of servers.

Drone is at the avant garde of a new breed of testing tools. It exhibits how we
can use cloud technologies to achieve greater control over a detailed test
suites, and thus build a stable and easy to maintain functional testing
platform. Do not be surprised if this nascent market gains focus in the next
year or two. Jenkins and the old-style continuous integration platforms cannot
hold a candle to this new breed of solution.

This article was originally for a print publication, but it never made it
that far.