The Software Tester’s Easter Egg Hunt

I recently came across a video clip of Paul Holland that was published last December by the Association of Software Testing. In the video (above), Holland makes the connection between a software tester’s day-to-day work and a good ol’ fashioned Easter egg hunt. What complicates this hunt, according to Holland, is that the “eggs” that testers are looking for are of:

Unknown Size

Unknown Shape

Unknown Color

Unknown Texture

Unknown Pattern

Unknown Quantity

And are hidden in unknown locations throughout an entire building

I love this analogy because it quickly summarizes the basic tasks associated with software testing while also conveying the impossibility of finding every bug in a piece of software. I also whole-heartedly agree with his overall point that skill-based testing is much more valuable than the ability to mindlessly create and run hundreds of scripted tests. The only issue here is that, in order to make Holland’s point a bit more realistic, you really need to add a few layers to this hunt:

The eggs aren’t always visible at first glance.

Each hunt lasts a finite amount of time.

After each hunt, someone else renovates the building and hides new Easter eggs.

When you include these factors in the equation, scripted tests can become an extremely valuable asset. Allow me explain by continuing with where Holland left off.

The Hunt

The reality is that you normally don’t have to choose whether you want to do a scripted or an exploratory search of each room. Instead, a scripting tool should simply be seen as a robot that’s there to aid in your search for Easter eggs. It’s not meant do all of your searching for you, it’s just there to handle the tedious and repetitive parts of the hunt.

Say you walk into a room and, knowing that it’s a popular spot for hiding Easter eggs, you decide to start by checking in the refrigerator. You open the fridge and look around, but you don’t initially see any Easter eggs. Just as you’re closing the fridge door, you catch a glimpse of something beside the milk that may or may not be an Easter egg. You know that you should open the fridge and look again, but you’re also aware that you have a time limit. Without the use of scripting, you’re somewhat limited two options here:

Spend time opening and closing the fridge until you confirm whether or not there’s an Easter egg next to the milk.

Ignore what you saw and move on.

The Role of Scripted Tests

Scripting opens up a third option when dealing with the scenario above. Instead of choosing between relentlessly scouring the fridge and ignoring it all together, you can enter the room and immediately tell your robot to go to the fridge, open the door, look next to the milk, and close the door. You can also tell your robot to repeat that task 1,000 times, and to report back to you every time an Easter egg appears.

While your robot is tediously checking the fridge, you’ve now got more time to consider how your normal searching guidelines apply to this particular type of room. This also gives you time to consider a few hiding spots that may be completely unique to that room, such as:

“This room has a fridge AND a bed? That’s strange. I should probably go look under the sheets, inside the pillow cases, behind the headboard, and on the floor underneath the bed. Maybe I’ll even cut the mattress open to see if any eggs are hidden inside of it.”

Without the help of that robot you could have spent hours trying to confirm the existence of an Easter egg appears to the left of the milk every 300 times you open the refrigerator. This only wastes precious time that could have been spent applying your own critical thinking skills to uncovering every nook and cranny in and around the bed.

The Next Hunt

Finally, that little robot could also be extremely handy months later when, after some renovations are made to the building, you go back for another Easter egg hunt. You now know that the robot specializes in opening and closing certain objects, so along with going back to the fridge (which now has three new eggs hidden in it) you also tell the robot to open and close the oven and each cupboard a few dozen times. Once again, this leaves you free to re-examine the bed and to dig into that new armchair that was just added in the far corner of the room.

Okay, so I’ve beaten this metaphor into the ground (sorry, Paul), but I do think it’s a discussion worth having. The testing industry will benefit greatly if more people follow Holland’s example and prioritize critical thinking over the ability to write test scripts. That said, it is equally important to recognize that scripting tools, when used in the situations they’re intended for, can be great time-savers that can enable more thoughtful, context-driven exploratory testing.

What do you think? Do you rely on any scripted testing in your organization? How has it helped or hurt your testing processes? Leave a comment below to give your feedback.

Comments

Also each bug may only appear when certain other conditions are in
specific states and when they are not in those states everything works
fine (pairwise and combinatorial bugs). Like I can’t use your comment
system with Chrome (“Disqus seems to be taking longer than usual. Reload?” – just forever, reloading…) but I can with Firefox.

The most apparent pairwise and combinatorial bugs can be caught with exloratory testing. Many may not be though.

Scripted testing is good to *check* specific settings for specific results. Doing a bunch of this programmatically is very useful. Doing some of it with a person looking for issues in specific test cases is wise.

Thinking this is all you have to do is very unwise. You need exploratory testing by a knowledgeable software tester (or if this isn’t possible then exploratory testing by a user proxy – this is not perfect but is much better than nothing) if you care about the quality of your software.

You’re absolutely right, John. I actually wanted to include another part of this post that talked about having the robot check the cupboards while I ran the blender on “pulse,” then “ice crush,” then “smoothie,” but I felt that I had already run Paul’s metaphor into the ground at that point.