Regressing to the Mean

With the popularity of the movie and book Moneyball, among other things, the principles of ‘advanced statistics’ seems to be everywhere you look in sports. As I read about these different methods of analysis, I keep reading of authors referring to people and teams “regressing to the mean”. To my eyes, it is mostly used as a blanket way of explaining the unexplainable. If a player goes on a hot streak, a drop in performance is regressing to the mean. If a team outperformed the historical trends last year, they should regress to the mean this year and do worse. It all seems like stopping 2 or 3 why’s short in a 5-Why, but this isn’t the forum to argue the specifics of advanced metrics in sports. However, as I have started to see the phrase “regressing to the mean” show up outside of those arenas I think it provides an interesting topic.

In the interest of time and space, I’ll keep the strawman simple here. The sports guys and gals have the concept mostly accurate from a big picture level. Over time, a process will show what its performance will be. The short term swings high and low are just normal variations that level out over time assuming no other significant factors intervene. From a human performance standpoint, what does that mean and how can we impact it?

People tend to have their own expectations of their performance. If they exceed their expectations at something, most are likely to perform worse at the same task in the future. There are a lot of factors at play in this such as differences in focus, preparation, expectations, complacency, and so on. The opposite is also true. When expectations aren’t met, the performance tends to increase. From a cultural standpoint, I tend to think this is one of the factors that help to propel companies like Toyota and other that are successful in Lean. There is a constant reinforcement of thought patterns like “ ’no problem’ is a problem” and chasing a True North state. This helps create an ongoing mindset of not performing at a high enough level. For the people that can function and even thrive in this environment, it develops a constant carrot to chase and keeps the organization as a whole from regressing.

I fully recognize that not everybody can function in that type of culture. That isn’t an indictment of them, just a recognition that all people are different and come to work with their own needs. I also recognize and have witnessed the farther end of the spectrum where the carrot of raised expectations turns in to the stick of failure. This doesn’t mean that we should stop identifying successes. It just brings an acknowledgment that reaching a certain level doesn’t mean that the climb is over. Individually, it can give us an opportunity to ask ourselves how comfortable we have gotten with our skills and performance. In turn, it’s an opportunity to look at those around us and try to understand how they view their successes and raising the bar of expectations. If we believe in the old adage that you are either getting better or you are getting worse, regressing to a mean isn’t really a viable option.