The Illusion of Skill

In his book Thinking Fast and Slow, Daniel Kahneman describes an "illusion of skill" where too much weight is attributed to skill rather than other factors like blind luck. He recalls a time he was invited to speak to 25 top investment advisers who had given him eight years worth of their spreadsheets to analyse.

Did the same advisers consistently achieve better returns year after year?

Did some display more skill than their peers?

To find the answers he looked at the correlations over the eight years but found no evidence of skill.

He concluded: "The stability that would indicate differences in skill was not to be found. The results resembled what you would expect from a dice-rolling contest, not a game of skill. The firm was rewarding luck as if it were skill."

One advisor went on the defensive when he heard this and said "I have done very well for this company and nobody can take that away from me." But if it's mostly down to chance, how much credit are you entitled to take for it? There is no doubt Kahneman's message was forgotten.

Kahneman talks about how we like nice stories that seem to fit the narrative over cold statistics. Emotion over logic. We are very poor at looking at the bigger picture.

How much chance do you think is involved in big business? Sports? How common is this illusion of skill? I'd be willing to bet it's very common.

Replies to This Discussion

There was a study on basketball players that questioned whether a player was on a "hot streak" i.e. they score four long shots in a row or something. The study found the players deemed hot would inevitably regress to the mean, thus there is no such thing as a hot streak. I can see how this might make sense in basketball but I don't know about other sports. It's all a bit confusing.

This is touched on in Fooled by randomness, Dice World, the Black Swan and The Halo Effect. In each case the workers believe "my job is essential" and "I am the best person to do it". Often the answers are "no" and "no". We don't even need to get into their obscene salaries.

"that CEO makes too much"

"yeah...but would you want a guy who would work for less to be at the helm"?

I lived in Portland, Oregon for about 35 years and wish I was back there. I love the Pacific Northwest, rain and all. Portland doesn't have and NFL team (they're more of a soccer town), so the Seahawks are my team.

I'm stuck in Cleveland. Came back to take care of my father. He died, but now I can't seem to get back there. I love the fact that Portand is such a walkable city It's compact. At the same time it has plenty of good restaurants and probably the best public transportation system in the country. It's the microbrew capital of the country and (don't tell Seattle) it has more coffee microroaster awards than Seattle (or any other city for that matter). It's a very clean city, too. It's been likened to a Scandinavian city in that regard.

During the course of 8 years investing for my own accounts, I had mixed results. For a few years I did very well, but sadly the party ended about 1983. It was clear that any claim I had to 'skill' was over rated, and that my results were mostly do to 'market. Having simply done a 'random sample' with a few 'good' hits!

My mother is very into investment literature, and I was nearly drawn in by it myself, in my more impressionable years, because each book she picked up seemed to have the answers to all questions you could ever want to ask about becoming rich and successful. While I would never say every one of these books is completely full of shit--there were good ideas in greater or lesser proportion to bad ones in each case-- it took both of us a while to realize that the entire industry of investment literature (and financial advice, and self-help lit and many other things) relies entirely on statistics. Here's how it works; if enough people write a diverse enough body of literature on a particular subject, a few of them are bound to stumble across occasional insights, or to seem prophetic of certain events, or to seem to have really captured the essence of a particular scenario at the time. I think primarily of one book, can't remember which, which predicted a financial crisis tied to the bursting of a housing bubble, published sometime in late 2007 I think. While I would never say his seeming prescience was simple chance, I am absolutely certain that things were not as simple as he painted them and that there was also a chance he could've been wrong and his book would never have risen from obscurity.

The same, I would argue, applies to almost any discipline where we use heuristic methods of detecting or constructing patterns (heuristics were a major subject of Kahnemann's early research). Belief in prayer is a fantastic example, and there is a wonderful take-down website devoted to the idea that the efficacy of prayer can be disproven by the fact that God has never seen fit to regrow the limbs of an amputee (www.whydoesgodhateamputees.com).

In my area of expertise, music, there exist circumstances in which skill is obvious and necessary, and some in which skill ceases to be an issue. In a college-level piano competition, for example, the level of technical mastery is very much an issue, usually because a student simply hasn't been alive and playing long enough to have the absolute assurance of an experienced performer. When comparing two great pianists, however, skill is often not an issue since they both have a very high degree of technical mastery, to the point where mistakes, if they happen, are very minor and easily smoother over. In the latter case, I (and many judges, to the entire music world's chagrin) find myself basing judgments on heuristic factors; a particular passage phrased or articulated a certain way might make or break it for me, or perhaps a tempo is not what I would prefer it to be. Even experienced listeners, at this point, have trouble deciding things on a basis of strict merit, hence, for example, the practice of auditioning symphony applicants behind a curtain (to guard against gender discrimination, particularly) and my practice of listening to live performances with my eyes closed (so I'm not distracted by whatever ridiculous faces and gestures the musician is prone to make).

The same can probably be said of almost any discipline. I have no doubt that, if I were placed in a call center worker's or a hedge fund manager's position, I would be severely outperformed by people who are more skilled in talking with people or at predicting the behavior of financial markets. This attests to those people's skill at what they do. However, get a couple of skilled telemarketers or bankers competing and the difference in outcomes is much more likely to be due to chance than skill. What this basically illustrates is that differences in skill (tend to) eliminate the variable of environmental factors, and that similar skill levels tend to allow environmental factors to be seen more clearly. None of that is to say that either skill or environmental factors play no role; both contribute to any endeavor.