When Expert Predictions Fail

The character of predictions

“Fact: Human beings love to predict the future. — Fact: Human beings are not very good at predicting the future. — Fact: Because the incentives to predict are quite imperfect — bad predictions are rarely punished — this situation is unlikely to change.” This is how an article by “Freakonomics” author Steven J. Dubner starts. Why is that and which predictions can we actually believe in?

In his article on the Freakonomics website, Steven J. Dubner elaborates a bit further on this. What does it mean that “the incentives to predict are quite imperfect”? Steven Lewitt, Dubner’s co-author, points out that whenever we make a crazy prediction and this prediction becomes true, we and others keep reminding everyone of it, whereas the many other crazy predictions that never became true are not talked about any more.

The article outlines a few studies on the quality of predictions. For example, we can see that experts often know little more than laypersons, and their predictions are often only slightly better than purely random ones. But they are not aware of this fact, but rather believe strongly in their expertise (this, by the way, was a rather famous study by psychologist Phillip Tetlock). Furthermore, we often misinterpret the verbalization of predictions. When something “could” happen, this means that it is likely to happen on a huge continuum, ranging from extremely unlikely to extremely likely. Moreover, when someone predicts an extreme outcome, we tend to overestimate that person’s accuracy in future predictions. Thus, one could consider bad predictions as an interaction of experts who are not quite as good at predicting something on the one hand and laypersons who would simply love to have a predictable world and thus interpret predictions more deterministically than they actually are on the other hand.

Why are there so many bad predictions out there? Economist Robin Hanson who is cited in the article thinks that one problem is that we better say nothing when we don’t know what to predict. However, what frequently happens is that e.g. journalists ask an expert for a prediction, and even if the expert has nothing to say, he or she will try to make a forecast just in order to say something. Robin Hanson is an advocate of what he calls a prediction market, i.e. a market in which only those who really have something valid to say do speak up, whereas the others say nothing (instead of making invalid predictions).

There was a whole radio show on this topic. Its transcript is available on the Freakonomicswebsite.

What does this mean? If you consider yourself to be an expert in a certain field, be aware of the fact that you might be overconfident. Constantly challenge your own predictions. If you are a layperson, be careful with predictions so-called experts make. The fact that they have been right once or twice does not mean that they will always be right. Take a closer look at previous predictions the expert has made and find other experts who make forecasts in the same area. Compare them and always remain critical. Remember: it is hard to make predictions, especially about the future.

About the Author

Dr Katharina Lochner is the former research director for the cut-e Group which was acquired by Aon in 2017. Katharina is now a researcher and lecturer at the University of Applied Sciences Europe in Iserlohn, Germany. In her role at cut-e, she applied the research in organizational and work psychology to real-world assessment practice. She has a strong expertise in the construction and evaluation of online psychometric tools.