Thursday, June 7, 2012

I was recently involved in a software release where some
rather unfortunate defects made their way to production. This resulted in some
(not unreasonable) tough questioning from our stakeholders.

One particular area they wanted to ask about was our testing
practices. Were we not testing our product the same way their user community
used it?

This highlighted an important disconnect in understanding
the variety of types of testing that a responsible, professional team would
usually employ to thoroughly test a product.

It was true that our testing wasn’t good enough for that
release. But there was much more to it than simply “testing the software in the
way users use it.”

In a bid to help them understand our weak areas (and what we
were doing to remedy those) I went hunting for one of those “test pyramids”
modeled on the same idea as the USDA food guide pyramid (http://www.nal.usda.gov/fnic/Fpyr/pmap.htm)

There are plenty out there, but I wanted to assemble my own
which is what you can see here (click to enlarge):

If I were to do this again I’d probably separate out a bit
more the different types of testing in the lower blue section for better
emphasis of each, but you can get the gist of things from it as is.

As you can see, I’ve classified the testing appropriate for
our product in quite a few different ways. Understanding what these all were, how they differed and why each different kind was necessary was (it became apparent) difficult for our non-technical
audience to easily understand.

Reflecting upon this later, I came up with the following
analogies that I hope helps make these different kinds of test more
understandable to non-developers (and even some developers...)

Why so many different kinds of test?

This too was a puzzle for some. The idea I came up with to explain this was that, just as a doctor may need a wide variety of tests to diagnose a patient, so too we need a variety of tests to check the "health" of our software.

The analogies below are all based around the idea of building cars. They might be a bet stretched, but I hope they're useful.

Unit tests

Inspecting each component in isolation. Confirm the
building blocks are fit for use in the bigger system (car).

Feature/acceptance tests

Checking all features behave as expected, e.g. does it
steer properly, doors and trunk open properly, seats adjust etc.

Performance tests

Driving around a racetrack, time trials, checking how
fast you can get from 0-60?