Tales from the Script

A number of roles ago the operations and developer folk were blessed with a
relatively inexperienced quality assurance department that were, to put it
kindly, focused on manual exploratory testing. They were, to a person, incapable
of writing any kind of automated testing, running third party tools or doing
anything in a reproducible way. While we’ve all worked with people lacking in
certain skills what makes this story one of my favourites is that none of us
knew they couldn’t handle the work.

The manager of the QA team, someone I’d never have credited with the
sheer audacity to pull off this long con, always came to our meetings
with an earnest face and excuses about the failure of “The Script”. We,
being insanely busy modern technical people, took this at face value;
how would you run all the regression tests without a script? “There was
a problem running the script”, “the newest changes to the script had
caused regressions” and similar were always on the tip of their tongue
and because the developers were under a lot of time pressure no deep
investigations were done. Everyone was assumed to be doing their best
and what a great QA manager they were in protecting their people from
any fallout from the failures. On it went, all testing was done via “the
script” and everything was again good. Or so we assumed.

In one of our recurring nightmare incident reviews, this one after
something we’d previously covered had come back for the third time, a
few of us began to get suspicious. We decided to build our own little
response team and do some digging for the sake of every ones sanity. Now,
this was before the days of GitHub and everyone being in one team of
sharing and mutual bonding. We knew we’d have to go rooting around other
departments infrastructure to see what was going on. Over the course of
the next few days the group targeted one of the more helpless QA
engineers and began to help him with everything technical he needed. He
had the most amazing, fully hand held, on-boarding the department had
ever seen and we, in little bits and pieces, began to pierce the veil of
secrecy that was the QA teams process.

One day, just before lunch, one of the senior developers involved in our
investigation hit the mother-load. The QA engineer had paired with them
on adding testing to “the script” for a new feature the developer had written
and suddenly he had a full understanding of the script and its
architecture.

It was an Excel spreadsheet.

It was a massive, colour coded, interlinked Excel spreadsheet. Each row
was a separate test path or page journey. Some rows were 40 fields of
references to other rows to form one complete journey. Every time we did
a release to staging they’d load up the Excel document from the network
share and arrow key their way through row upon row of explicit
instructions. Seeing it in use was like watching an insane cross between
snake and minesweeper. Some of the cells were links to screen grabs of
expected outputs and page layouts. Some of them had a red background to
show steps that had recently failed. It was a horrific moment of shared
confusion. A team of nearly forty testers had ended up building this monstrosity
and running it for months. It was like opening up a word doc and having Cthulhu
glare back at you. So we did the only thing we could think of, went to lunch and
mostly sat in stunned silence.

And I almost forgot the best part of the story, the Excel spreadsheet?
It was named “The_Script.old.xls”