From One Expert to Another: Dale Emery

In this From One Expert to Another column, Payson Hall interviews Dale Emery about a recent talk he gave on the "Test Automation Zombie Apocalypse." Here, Dale gives tips on test automation and what makes "zombie ideas" dangerous.

Payson Hall: Dale, you recently gave a talk about the "Test Automation Zombie Apocalypse." What was the origin of that talk?

Dale Emery: Lee Copeland invited me to fill in for a conference speaker who had dropped out, for a talk about test automation. We created the title first, and it really seemed to resonate with something I had been mulling over in my head: the test automation tradeoffs that people make and the natural human tendency to make tradeoffs that favor short-term results.

Like regular zombies, test automation zombies are relentless and infections, and they eat your brains. Unlike regular zombies, test automation zombies don’t look like people. They look like ideas. Every one of these zombie ideas comes from a good intention, and every one is sometimes the right thing to do.

Payson Hall: What are some examples of test automation "zombies"?

Dale Emery: Some movie zombie stereotypes seem popular and recur often—for example, zombie woman in a business suit at the shopping mall, zombie Jethro from the gas station in overalls, and zombie who waits in the dark for unsuspecting young people. Like cinematic zombies, there are some cliché test automation zombies I have observed in different client contexts. Let me describe a few of these.

The one that scares me most is the record and playback zombie. This zombie is the idea that all you have to do is run through your manual test suite, which you were going to do anyway, and you end up with automated tests. The appeal here is obvious: You get a huge suite of automated tests almost for free, and then you can run them as often as you want.

The first problem appears almost immediately: The tests are brittle. The slightest change in system implementation makes recorded tests fail for the wrong reasons. Then you notice the second problem: Recorded tests are utterly unmaintainable.

And then the worst problem of all appears: People attribute the maintenance nightmare not to the zombie idea of record and playback, but to test automation in general. So, they abandon test automation altogether.

Also common is the test automation group zombie. This zombie is the practice of assigning test automation to a dedicated team of test automators. The appeal is that we can keep developers focused on writing new code instead of writing and maintaining automated tests. The danger is that test automation inevitably lags development, so feedback from testing is delayed in a way that significantly reduces its value. A further danger is that we often assign test automation to less-skilled programmers, which results in brittle tests that are costly to maintain.

There’s also the code coverage zombie, which fools us into believing that high code coverage means that our application is well tested. And the fixed wait zombie, which invites us to sprinkle delays through our tests to accommodate variations in response times but makes our tests take longer and longer. And the chewy GUI zombie, which invites us to automate only through the system’s graphical user interface, making automated tests vulnerable to changes in the most volatile part of the system—even when those changes are utterly irrelevant to the tests.

So far, I’ve identified fifteen test automation zombies in the wild, and the catalog is growing. [1] They’re everywhere.

Payson Hall: It sounds insidious that some of these might be reasonable ideas in some contexts. Is that what gives them life?

Dale Emery: Yes. In every case, the desire that gives birth to the zombie is perfectly reasonable. We want to keep costs low. We want confidence in our tests. We want to minimize our automated tests’ susceptibility to normal variability in system response times.