Goat entrails, tea leaves, meteorologists, and Nobel-winning economists: On the art of prediction and the language of forecasting

Experts who make predictions better than other experts have a different style of thinking and, indeed, demonstrably use language differently. That’s one of the points explicated in the lead article for the Cato Institute’s online journal, “Overcoming Our Aversion to Acknowledging Our Ignorance,” by Dan Gardner and Philip Tetlock.

Gardner and Tetlock are convinced that the ability for improvement in predictive ability is crucial to human society and is currently systematically underacknowledged. They go so far as to point to the fact that “widespread lack of curiosity — lack of interest in thinking about how we think about possible futures — is a phenomenon worthy of investigation in its own right,” and advocate ignorance of the possibility for the degree of possible systematic improvements in the techniques of prediction. All this, but they don’t stop to explore in detail the nature of the differences in styles of thinking, and how it is possible that some thinkers appear capable of higher levels or more sophisticated degrees of cognition of the future than others.

There’s a massive ghost in the essay, and it is a blindness — almost certainly willful at least in part — of the realms of psychic intuitions, divination, and the distinct uses of language at different stages of consciousness development. Indeed, once the spook of clairvoyance reared its head in the essay by way of a quotation of James Clapper:

When governments the world over were surprised by this year?s events in the Middle East, accusing fingers were pointed at intelligence agencies. Why hadn?t they seen it coming? ?We are not clairvoyant,? James R. Clapper Jr, director of national intelligence, told a hearing of the House intelligence committee. Analysts were well aware that forces capable of generating unrest were present in Tunisia, Egypt, and elsewhere. They said so often. But those forces had been present for years, even decades. ?Specific triggers for how and when instability would lead to the collapse of various regimes cannot always be known or predicted,? Clapper said.

In other words, when those charged with responsibility to make predictions bump up against their fallibility and ignorance, they throw up their hands in futility and blame their lack of clairvoyance, as if to say they are not superheroes who can leap tall buildings with a single bound. But is clairvoyance really an incredible superpower only to be found in comic books and Hollywood movies?

Gardner and Tetlock have done a superb job of describing the importance of acknowledging ignorance, but by neglecting to explore topics that they probably deem too woo-woo or fringe, they simply reinforce the conventional resistance to looking into some of the more peculiar and seemingly inexplicable features of mental life. They also, as it happens, neglect a solid body of philosophical discussion and psychological research on consciousness.

There’s a bromide which informs young academics that if they want to research extrasensory perception (ESP), they best wait until after they get tenure. One gets the decided impression that Gardner and Tetlock are like junior faculty presenting the strange case of people with different styles of thinking to senior faculty, and praying with fear and trembling that nobody notices that they are tiptoeing on the borders of occult knowledge.

Just what sort of different styles of thinking do they describe, and how are they connected — if at all — to clairvoyance? In one of the essay’s most important sections, they write:

In the most comprehensive analysis of expert prediction ever conducted, Philip Tetlock assembled a group of some 280 anonymous volunteers?economists, political scientists, intelligence analysts, journalists?whose work involved forecasting to some degree or other. These experts were then asked about a wide array of subjects. Will inflation rise, fall, or stay the same? Will the presidential election be won by a Republican or Democrat? Will there be open war on the Korean peninsula? Time frames varied. So did the relative turbulence of the moment when the questions were asked, as the experiment went on for years. In all, the experts made some 28,000 predictions. Time passed, the veracity of the predictions was determined, the data analyzed, and the average expert?s forecasts were revealed to be only slightly more accurate than random guessing?or, to put more harshly, only a bit better than the proverbial dart-throwing chimpanzee. And the average expert performed slightly worse than a still more mindless competition: simple extrapolation algorithms that automatically predicted more of the same.

Cynics resonate to these results and sometimes cite them to justify a stance of populist know-nothingism. But we would be wrong to stop there, because Tetlock also discovered that the experts could be divided roughly into two overlapping yet statistically distinguishable groups. One group would actually have been beaten rather soundly even by the chimp, not to mention the more formidable extrapolation algorithm. The other would have beaten the chimp and sometimes even the extrapolation algorithm, although not by a wide margin.

One could say that this latter cluster of experts had real predictive insight, however modest. What distinguished the two groups was not political ideology, qualifications, access to classified information, or any of the other factors one might think would make a difference. What mattered was the style of thinking. [Emphasis mine.]

One group of experts tended to use one analytical tool in many different domains; they preferred keeping their analysis simple and elegant by minimizing ?distractions.? These experts zeroed in on only essential information, and they were unusually confident?they were far more likely to say something is ?certain? or ?impossible.? In explaining their forecasts, they often built up a lot of intellectual momentum in favor of their preferred conclusions. For instance, they were more likely to say ?moreover? than ?however.?

The other lot used a wide assortment of analytical tools, sought out information from diverse sources, were comfortable with complexity and uncertainty, and were much less sure of themselves?they tended to talk in terms of possibilities and probabilities and were often happy to say ?maybe.? In explaining their forecasts, they frequently shifted intellectual gears, sprinkling their speech with transition markers such as ?although,? ?but,? and ?however.?

Later in the essay they imagine an institutional system (an “analytical center”) that might be capable of tapping into the power of persons with a style of thinking demonstrably superior in predicting the future. They write:

Imagine a system for recording and judging forecasts. Imagine running tallies of forecasters? accuracy rates. Imagine advocates on either side of a policy debate specifying in advance precisely what outcomes their desired approach is expected to produce, the evidence that will settle whether it has done so, and the conditions under which participants would agree to say ?I was wrong.? Imagine pundits being held to account. Of course arbitration only works if the arbiter is universally respected and it would be an enormous challenge to create an analytical center whose judgments were not only fair, but perceived to be fair even by partisans dead sure they are right and the other guys are wrong. But think of the potential of such a system to improve the signal-to-noise ratio, to sharpen public debate, to shift attention from blowhards to experts worthy of an audience, and to improve public policy. At a minimum, it would highlight how often our forecasts and expectations fail, and if that were to deflate the bloated confidence of experts and leaders, and give pause to those preparing some ?great leap forward,? it would be money well spent.

So what the essay describes is evidence that persons whose structure of consciousness (i.e., style of thinking) is systematically analytical, possessing a preference for incorporating diverse elements into harmony, tolerant of complexity, comfortable with uncertainty, possessed of a certain flexibility of self-sense, and an ability to take multi-dimensional perspectives. They used language differently from less effective predictors in empirically and objectively verifiable ways.

None of this is at all surprising when seen from an Integral philosophical point of view, such as the meta-theory of consciousness of Ken Wilber or Allan Combs. These scholars have painstakingly documented the differences between interior and exterior structures of consciousness at pre-conventional, conventional, and post-conventional levels. I believe it is likely that the multi-perspectival and complex qualities of superior predictors noted by Gardner and Tetlock are characteristic of late-stage structures of consciousness, and the simplistic and certainty-minded views of inferior predictors are found in early stages of mind.

Psychological investigators and theorists such as Jane Loevinger, Susanne Cook-Greuter, and Terri O’Fallon have made related points by investigating late stages of ego-development. Their models of psychological development have begun to glimpse rare realms of consciousness development once thought of as woo-woo, occult, or mystical. One of their empirical findings is that people at different stages of ego-development use language differently.

For example, Cook-Greuter found that individuals at a conventional (specifically Conformist Stage 3/Diplomat) use short, stereotypical phrases, love clichés, and exaggerate positive affect. What do you know? Birds of a feather flock together. In contrast, persons at an early post-conventional stage (specifically Individualist or Pluralist Stage 4/5) begin to replace “or” and “but” with “and” and show increased use of the phrases “even though” and “despite.”

One of the features of some of the highest levels of awareness, according to investigators of advanced ego-development, is increased receptiveness to non-rational modes of cognition and an ability to create complex maps or worldviews which make room for awareness outside of space and time. Regarding one of the most advanced and rare stages, Construct-Aware Stage 5/6, Cook-Greuter writes:

By turning further inward, Construct-aware persons start to see through their own attempts at meaning making, and become aware of the profound splits and paradoxes inherent in rational thought. In many ways, they individually rediscover the Korzybski?s notion (1948) that ?the map is not the territory.? The linguistic process of splitting into polar opposites and the attending value judgments can become conscious. Good and evil, life and death, beauty and ugliness may now appear as two sides of the same coin, as mutually necessitating and defining each other. Moreover, the constant judging of what is good and what is not creates much of the tension and unhappiness so prevalent in ordinary waking consciousness. Construct-aware individuals generally have a dynamic and multi-faceted understanding of human nature and the complexities of human interaction. They want to face their own profound need for theories and explanations. They hope to unearth the limits of the rational mind, and to unlearn their automatic, conditioned responses based on memory and continuous, everyday cultural reinforcement. (“Ego Development: Nine Levels of Increasing Embrace,” unpublished but available at www.cook-greuter.com)

She sometimes speaks of such advanced realizations of ego-development as an “Alchemist” or “Magician” stage of awareness, alluding to the ability of this style of thinking to sometimes befuddle the rational mind with an uncanny perceptiveness and an ability to generate and perceive transformative trends. It would not be at all surprising if modes of thinking that Gardner and Tetlock might call psychic, clairvoyant, or divinatory involve these higher realms of awareness in some way. Persons with styles of cognition more adept at predicting and controlling the future have always been among us, and it stands to reason that ancient and medieval cultures would have broadly thought of their abilities as magical in nature.

And so we turn to the essayists’ conclusion: “Humility is in order, or, as Socrates said, the beginning of wisdom is the admission of ignorance.” Let us wholeheartedly agree with Gardner and Tetlock’s call for greater attention to the arts of prediction, let us embrace their insistance on attempts to harness accurate forecasters’ abilities for establishment of public policy, and let us insist as they do on humble awareness of what we don’t know.

But there is much more that we know about cognitive styles with greater ability for prediction than these essayists tell us, if we are humbly willing to look at the evidence identified by Integral theorists and advanced research in ego-development, even if it suggests challengingly that styles of thinking are not all created equally, and our own may not be (indeed, almost certainly is not) the most advanced possible.

It is not merely our aversion to acknowledging our ignorance, per Socrates, that must be overcome, but our aversion to acknowledging our limited stage of consciousness, per Wilber. If we do so, who is to say that we will not find more truth about the claims of psychics and clairvoyants than is to be found in the conventional discourse?