Top Tabs

Thursday, April 1, 2010

Calibration

In a previous post, we saw how humans have an overconfidence bias, and how that can play havoc on financial forecast estimations. What is not immediately clear, however, is the increasing role overconfidence plays the more knowledge one acquires. That is, as expertise rises, so does overconfidence, resulting in the fact that the people with the most knowledge are likely to be the most miscalibrated, which can result in detrimental effects.

But knowledge alone does not doom someone into being overconfident. James Montier argues that it's a lack of feedback that pushes people to be overconfident. To demonstrate this, Montier cites a study performed by Scott Plous that compares the calibration of weathermen against that of doctors. Weathermen were asked to predict the weather, while doctors were asked to diagnose patients (based on case notes). Both were asked to provide confidence intervals, so that their calibrations could be measured.

Interestingly, weatherman were well calibrated, while doctors were very poorly calibrated. Montier argues that this is because weather forecasters benefit from the fact that they receive immediate evidence of their abilities as forecasters, while doctors do not.