Menu

Why you can’t test a product and why you will never be able to verify anything that matters

I make two pretty drastic claims in the title– You can’t test a product! – You will never be able to verify anything that matters

In this post I will explain what I mean with those statements and provide my arguments for why I believe it is so.

Our understanding of the world is full of simplifications. I like to think of our understanding of the world as very large set of mental models, all more or less connected with each other. All helpful, but also more or less incorrect. The testing world is no exception to this phenomena. Models like “testing the product against the spec” can be very helpful. But to evolve our understanding we need to question our understanding and come up with something that’s more accurate of what is actually happening. A more accurate model might be more complex and a bit more difficult to work with in day to day testing. But it helps us understand how our simplified models sometimes can get us into trouble.

You don’t test the product, you test your understanding of the productA simple model of testing could be that you test the product against a specification (or some other oracle of your choosing). Basically comparing the product against some spec. But it’s not quite that easy. What is closer to the truth is that you compare your understanding of the specification with your understanding of the product. The “compare” is a pretty big simplification but we return to that later. For now we simply want to compare our understandings. There are many ways one can improve understanding of specifications but that’s out of scope for this article. Here we focus on improving our understanding of the product. So what do we do? Look at the specification? No! Looking at the specification would not build our understanding of the product, it would build understanding about the spec. Ahh, so we start testing the product! No!

Wtf? Not test the product!?!

Nope. The way I see it we start off by guessing about the product. More or less consciously we theorizing about the product. In a fraction of a second our brain has build the first model of the product. Then we start testing, not the product, but we test our understanding of the product. A technical empirical investigation to reveal flaws in our understanding of the product. Which investigations or tests you choose to perform is governed by a set of different models and emotions. I hope to write an article on those models soon also.

The rock
Imagine a big rock on a flat surface. Is it heavy? Your previous experience with rocks of that size tell you immediately that rocks of that size is heavy. The reasonable thing to do is to guess that it is heavy. You’re a tester, you decide to test. Based on your guess that this is a heavy stone you decide to lift with your knees instead of your back. It was a heavy stone. Now, could you have made that test without having no idea of what to expect? Even with minimal experience with stone lifting, any reasonably sane person who encountered at least a few stones would expect it to lie still as you approach it. Because that’s what stone on flat surfaces do, right? A stone with anthropophobia is unheard of. Your model of stones says that they don’t roll off when approached by humans. When approaching products we have all kinds of ideas about what we “know” about it based on previously experiences with similar products. Even with no experience of anything like it we still expect it to conform with basic models of nature laws.

A reasonable person makes good guesses about stuff. A good tester recognises that a large part of what we think we know about stuff is guesswork, and tries to challenge those guesses by empirical investigations aka tests. If our investigation is coherent (another model) with our understanding have we verified that what we think we know is actually true? Nope.

Once upon a time there was a splendid theory about sunrise and sunset. Sun rises in the east in the morning and set in the west in the evening. A good model of how the world worked. Partly that it excluded things that can not happen (a model that explains everything explains nothing) and partly because it was very testable. This theory was verified over and over again. Model mapped extremely well to observations and all was fine until one guy decided to explore the northern part of the globe. Imagine his surprise when he realized that the sun didn’t set at all way up in the north. How many white swans do you need to observe before one can verify that all swans are white?

You will never be able to verify anything that mattersCan we know that all stones on flat surfaces lie still when approached with humans? No, we can’t. Even if we test all stones in the universe we can’t be sure. Maybe something changed since we tested? Maybe we approached the stones from the wrong direction. We can’t know. And that is actually just fine. Even if we were wrong, it would not be a disaster, albeit a very exciting discovery. So what can we know? Only that ONCE when we approached a CERTAIN stone in EXACTLY this way it didn’t move in any way that I was able to observe. We can’t verify anything that actually matters.

So what can we do?We can build as good understanding of a product as possible by guessing, thinking critically if our guesses make sense and testing them as ruthlessly as we possibly can. By testing we can remove the faulty bits in our understanding and replace them with something that hopefully that it stands up better to critical analysis and harsh tests.

A very simple yet more helpful and accurate model
This is a simple high level model of how I see testing. Is it a good model? Well, it helps me think a bit deeper about testing and to me it represents more accurately what’s really going on when I test stuff. It’s wrong, but to me it is helpful. It helps me realize the complexity of testing. And from this simple model I can add more complicated models.

Conclusion
As I see the more traditional view of testing it actually consists of many different parts: Testing our product model to refine it, testing all of our other models about other stuff in the world like specifications. And refining all the models that helps us make sense of what these models of the products and it’s context is telling us. All this to be able to act in a way to achieve the goals that we want to achieve as testers.

I hope that you think this model is okay but decides to come up with a better, more accurate, more helpful one. And that you share it with with the world so that we can build on each other models. To verify the correctness of this model is impossible, just like verifying anything else. But it is also a really crappy model of how testing works since there is no way refute it. It just the way one tester represented his mental model about testing at one point it time.