A Fable: Thinking Out of the Box, part 2

[This is part 2 of my previous blog post “A Fable: Thinking Out of the Box”please read part 1first]

A week has passed since your “out of the box” experience and you haven’t stopped thinking about it since. Every day you have been thinking, pondering, quietly reflecting on the very idea that there is professional wisdom “out there” that not only don’t you understand, but seems to be in direct contradiction to everything you’ve ever known about QA. Determined, you decide to explore this a little more.

Once again, you rise up above your box, hovering over the landscape. Floating back to that strange-idea neighborhood, you hope that this time you might be able to meet an owner of one of those boxes.

Good news! As you arrive, you see a scholarly looking bearded man next to the oldest looking worn-out box.

“Greetings, Tester!” he warmly welcomes you as he looks up from his smartphone. “I was just updating my Twitter feed. You know, to stay relevant in this testing industry you need to keep on top of the latest innovations and controversies. There are amazing conversations about software testing happening in the #testing hashtag every day. For instance, this #stop29119 petition I was just reading about. But enough about me. I see you came from the other side of town. How can I help you?”

“Well,” you begin, “you see, I was just wondering how it could be that you and your box look so wise and experienced, yet, I saw that your box contains some strange sounding wisdom. So I wanted to understand what that’s all about.”

“Very good! Please give me an example so I can help you”, he says.

“OK, well how about when you say, ‘Regression testing is a waste of time’…how can you say such a thing? We need to run our regression suite each time the product is updated to make sure that no new defects have been introduced!”

The bearded man smiles. “A lot of people have trouble with that one”, he replies. After pausing for a few moments, stroking his beard in deep thought, he continues. “Think of it this way. If you’re concerned that there may be a problem with the updated build, then of course you should test it. But that’s not what I mean. I’m talking about re-running the exact same tests in the exact same way each time. What new information will you learn about the product if you do that? Maybe a little, but whatever you learn it’s hardly worth the effort.”

“Yes, but don’t you have to ensure there are no defects?”, you ask.

“Look. Are you familiar with the minefield analogy of testing? Think about walking through an open field that has explosive mines planted all around it in random spots. You see the footprints of the person head of you, who has already gone through the minefield. If you want to avoid stepping on the mines and blowing yourself up, you’re best off walking exactly in the same path, footprint for footprint, of the person that went before you, without any variation whatsoever. You would be very careful to do that exactly, so that you’ll make it through without anything exploding around you. The same thing is true when bug hunting. If you want to AVOID finding any new bugs about the product, then you should follow exactly, without variation, the same tests that have been run before. But when testing, we DO want to find new information. So when we run a new test we will get that new information, but when we run an old test it’s more likely that we won’t.

“Pay attention”, he continues, “because this is the key point”. “I’m not saying that all regression testing is inefficient. If you run the same tests a little harsher each time you will learn something from your experiment. Remember, testing is scientific experimentation, and if you want to learn how to run great tests, you should learn how to design great experiments. Just like a scientist – because that’s what expert testers are…scientists. So, for example, if you use more challenging test data than the last time, or try different features in different combinations or sequences each time, you will learn something and you may find new information to report to your stakeholders. Isn’t that what you’d like to do? Does this help you understand?”

You’re overwhelmed. Somehow this makes sense, but you’re still confused. This is a completely different way of thinking about testing than you’re used to. It’s a different perspective…no…a CONTRARY perspective than what you’ve known up until now. Mixed emotions are flowing through you: confusion, excitement, fear, skepticism, anticipation. “I guess this makes sense”, you finally say. “But I need some time to think this through.

“I’m so glad you said that!” the bearded man exclaims with a huge smile on his face. “Never ever believe anything anyone tells you without thinking it through for yourself! That’s one of the first rules of being an expert. You are well on your way!”

“Thank you. Hey, it’s late, and I need to get back. Can I come back here sometime? I have a feeling that there’s a lot more I can learn from you.”

“Are you kidding? Of course you can…I’d be thrilled if you came by again.”

“Thanks again, take care.”

You head back home. You’re looking forward to learning more about this interesting new outlook about testing. You get home to your box, and thoughtfully, open a Twitter account.