Climate seers as blind guides

The science of forecasting is complex. After 50 years spent studying the issue, I have found there is plenty of experimental evidence that in complex, uncertain situations, experts cannot forecast better than those with little expertise. In 1980, MIT Technology Review published my “Seer-sucker Theory”: “No matter how much evidence exists that seers do not exist, suckers will pay for the existence of seers.” Since 1980, research has provided more evidence for this surprising theory, especially Philip Tetlock’s 2005 book, “Expert Political Judgment.”

Forecasts of dangerous man-made global warming rely heavily on expert judgments. Is the global warming alarm movement another example of the seer-sucker phenomenon? If so, what is the scientific approach to climate forecasting?

In the 1990s, I organized an international group of 39 scientists from various disciplines to summarize principles for a scientific approach to forecasting. The principles are based mostly on experimental studies on what works best in given situations. Some, such as the principle of full disclosure, are based on commonly accepted standards. The findings were translated into a list of 139 scientific principles and published in the book “Principles of Forecasting” in 2001. The principles are available at forecastingprinciples.com, and they are revised as new evidence becomes available. This site includes a freeware package that allows anyone to audit forecasting procedures.

In 2007, I along with Kesten Green from the University of South Australia, published an audit of the procedures used by the U.N. Intergovernmental Panel on Climate Change (IPCC) to produce “projections” of global warming. The IPCC authors used computer projections derived from some scientists’ expert judgments. They call the projections “scenarios” (i.e., stories). As the authors admit, they are not forecasts, yet they are used as such. The audit showed that when the IPCC procedures are assessed as if they were forecasting procedures, they violated 72 out of 89 relevant scientific forecasting principles.

What does scientific forecasting tell us about global temperatures over the next century?

In 2009, Mr. Green, Willie Soon of the Harvard Smithsonian Center for Astrophysics and I conducted a forecasting validation study using data from 1850 through 2007. We showed that a simple model of no trend in global mean temperatures for horizons of one to 100 years ahead provided forecasts that were substantially more accurate than the IPCC’s 0.03 degrees Celsius per year projections. For horizons of 91 to 100 years, the IPCC’s warming projection had errors 12 times larger than those from our simple model. Our own forecasting procedures violated only minor evidence-based principles of forecasting, and it did not rely on expert judgment about the trend. Scientific forecasts since that 2009 paper, described in our latest working paper, assess those minor deviations from the principles, and the results support our earlier findings.

Have there been similar cases in the past where leading scientists and politicians have concluded that the environment faces grave perils? In an ongoing study, we have identified 26 alarmist movements that were similar to the current man-made global warming alarm (e.g., population growth and famine in the 1960s, and global cooling in the 1970s). In all cases, human activity was predicted to cause environmental catastrophe and harm to people. Despite strong support from leading scientists, none of the alarmist movements relied on scientific forecasting methods. The government imposed regulations in 23 of the 25 alarms that involved calls for government intervention. None of the alarming forecasts turned out to be correct. Of the 23 cases involving government interventions, none were effective, and 20 caused net harm.

Policy on climate change rests on a three-legged stool of forecasts. First, it is necessary to have valid and reliable scientific forecasts of a strong, persistent trend in temperatures. Second, scientific forecasts need to show that the net effects of the trend in temperatures will be harmful. Third, scientific forecasts need to show that each proposed policy (e.g., a policy that polar bears require special protection because of global warming) would provide a net benefit relative to taking no action. A failure of any leg invalidates policy action.

Since 2007, we have searched for scientific forecasts that would support the three- legged stool of climate policy. We have been unable to find a single scientific forecast for any of the three legs — the stool currently has no support.

Two ways to encourage unity on the climate change issue would be to insist that forecasts be provided for all costs and benefits, and that all forecasting procedures abide by scientific principles. If validated principles are not included in the current forecasts, they should be added. Until we have scientific forecasts, there is no basis for unified action to prevent global warming — or cooling. Rational climate policies cannot rely on seers, no matter how many of them, how smart they are or how much expertise they possess.

J. Scott Armstrong is a professor at the University of Pennsylvania and author of“Long-Range Forecasting” (Wiley-Interscience, 1985).

45 thoughts on “Climate seers as blind guides”

The Welsh politician Aneurin (Nye) Bevan who was largely responsible for the establishment of the British National Health Service in 1948 once said “Why look into the crystal ball when you can read the book?”
Perhaps climate scientists might learn more by reading J. Scott Armstrong’s book “Principles of Forecasting” than they would by gazing into their crystal balls or studying the output from their models as they would describe it.

I remember the 2007 report. Gavin’s response was to the effect of “that doesn’t apply to us because our forecasts are physical models.” (Seriously, he really said that.) The other argument advanced was “we can hindcast, so the models must be right.” (Face, meet palm.) Oh, and of course there was wide recitation of the evergreen mantra “they’re not climate scientists.”

Roy says: “The Welsh politician Aneurin (Nye) Bevan who was largely responsible for the establishment of the British National Health Service in 1948 once said “Why look into the crystal ball when you can read the book?””
In Bevan’s case, “the book” probably meant Das Kapital.

“…experts cannot forecast better than those with little expertise.”
About 30 years ago I was at a cocktail party in Colorado. When I met a guest who worked for the National Weather Service in Denver I just had to pick on him about the accuracy of their snow predictions for the front range. After a few barbs, he looked at me a said:
“Joe, to be honest, everyone in Denver has about the same success rate. In fact, if you screw up anywhere else in the country, they transfer you to the Denver office because nobody can really predict the weather for the front range.
” One winter we even tried a dart-board. We set the background to represent a “trace” and then filled it with clouds representing 1″ to 3″, 3″ to 6″, 6″ to 12″, etc. We set the size of the clouds and total area of each group of clouds relative to the board equal to the historic probability of each event.
“Anytime we suspected snow that winter we had someone throw a dart at the board. Guess what…. the dart board was as accurate as anyone else in the office.”
At least at that time, there were no ‘experts’ in front range weather though you occasionally met someone claiming to be.

If there is global warming, then we need the IPCC.
If there is no global warming, then we don’t need the IPCC.
Is there global warming?
Let’s re-phrase as if speaking to someone from the IPCC.
If there is global warming, you have a job.
If there is no global warming, you don’t have a job.
Is there global warming?

This relates to my personal observation from 30 years working with large scale software that people have an irrational faith in what computers tell them. To the point that telling them the computer is wrong, as I have had to do many times, produces what I can only call cognitive dissonance.
Not everyone suffers from this, but from my observations, a large percentage do.
IMO, that climate predictions/projections come from computer models, rather than say a bunch of calculations on a page, is a significant factor in their unquestioning acceptance by so many people.

Excellent post. I particularly enjoyed the application of the three-legged stool to this problem. You cite an “ongoing” study of 26 alarmist movements and governments actions to address them – I would be interested in seeing more detail when it becomes available.
Regarding seers, there has been some excellent research in the social sciences on perceptions of expert predictions. I recall one study in particular where subjects were provided a number of tokens, and then asked to bet on the outcome of a coin toss. Before tossing the coin they were given the option to purchase a prediction of the outcome, which was sealed in an enevelope, for the price of one token. After the coin toss, even if they did not purchase the prediction, they were asked to open the envelope to view the prediction. This was continued for three rounds. The interesting part is participants were much more likely to use a token to purchase a prediction on the third round if the prediciton for the first two rounds were correct. This despite the fact that the predictions were provided in advance in numbered sealed envelopes, they performed the coin toss themselves, and used a standard coin they were asked to bring with them. So even though you could explain to any rationale person that there was zero value in the prediction, it gained pereceived value based on past performance. I am struggling to find the link to the reference but will repost here if I can dig it up.

J. Scott Armstrong is a professor at the University of Pennsylvania
The well respected University of Pennsylvania is located in Philadelphia, Pennsylvania; think of Benjamin Franklin, rather than of that other well known place in central PA known for a scandal in the football program, and also the Earth System Science Center that has a famous “climate scientist” as director. It started as an Agricultural school (Farmers’ H. S.) in a no-where spot now called University Park as part of the municipality of State College. In the past, the area has been called “Happy Valley” – that’s worked out well!

Found the link to the study mentioned in my previous comment. “Why Do People Pay for Useless Advice? Implications of Gambler’s and Hot-Hand Fallacies in False-Expert Setting”http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2066980
The abstract:
We investigated experimentally whether people can be induced to believe in a non-existent expert, and subsequently pay for what can only be described as transparently useless advice about future chance events. Consistent with the theoretical predictions made by Rabin (2002) and Rabin and Vayanos (2010), we show empirically that the answer is yes and that the size of the error made systematically by people is large.

My favorite passage:
“The government imposed regulations in 23 of the 25 alarms that involved calls for government intervention. None of the alarming forecasts turned out to be correct. Of the 23 cases involving government interventions, none were effective, and 20 caused net harm.”

“The government imposed regulations in 23 of the 25 alarms that involved calls for government intervention. None of the alarming forecasts turned out to be correct.”
I betcha they are still active regulations to this day.

A prediction or estimate of an actual value in a future time period (for time series) or for another situation (for cross-sectional data). Forecast, prediction, and prognosis are typically used interchangeably.

Hmm, that doesn’t get us far…
So what do we do, guess? Read the entire dictionary, or a book, just to tell what a few paragraphs say? There seems to be some emphasis on the IPCC’s scenarios (stories), which give different answers for different conditions, e.g., for various possible amounts of future CO2 emission. Since we get a variety of answers for different assumptions, we can’t be getting just one “prediction or estimate of an actual value,” is that why they’re not forecasts? Do we have a real, valid “forecast” for each of these scenarios, but no forecast at all when you put them all together, or is the distinction something else entirely?
If the IPCC procedures violate 72 out of 89 relevant scientific forecasting principles, what are some examples of these? Preferably simple examples, with obvious importance, but at least, how about a starting point? The point about 26 alarmist movements, with 23 interventions and 20 cases of net harm, is similarly vague.
This article seems to make some interesting points. I’ll look forward to seeing these points elaboratated, and to seeing how they are supported.

If it walks like a duck and quacks like a duck….. the use of terms to obfuscate the actual intent is a well known bureaucratic trick. Unfortunately the public keeps falling for it. Call it “story” or “scenario” but treat it like a prediction. Call illegal aliens “undocumented”. Call a war a “peace keeping action”. Shakespeare didn’t go far enough. After you hang all the lawyers make sure you get the bureaucrats.

My forecast for Chicago, from NOAA, begins:
THE CONCERNS WITH THE FORECAST CENTER ON THE PRECIPITATION EVENT
LATE TONIGHT THROUGH THURSDAY EVENING.
A CHALLENGING FORECAST FOR THURSDAY AS GUIDANCE SHOWS QUITE A
SPREAD…NAMELY DUE TO THERMAL PROFILES AND SOMEWHAT DUE TO
TIMING. HAVE FOLLOWED A SLIGHTLY COOLER SOLUTION ALOFT THAN THE
NAM OR SREF MODELS WHOSE SOLUTIONS OF KEEPING THINGS AS RAIN FOR
THE ENTIRE EVENT JUST SEEM TOO LONG. THAT SAID…AT THE SURFACE
HAVE GENERALLY LEANED TOWARD A MILDER SOLUTION BECAUSE OF THE
ONGOING TRENDS AND A LIMITED SNOWPACK SOUTH OF I-88. THAT PROVIDES
MUCH LESS SNOWFALL AMOUNTS THAN WHAT IS BEING PROVIDED FROM THE
GFS WHICH JUST SEEMS TO BULLISH ON SNOWFALL ACCUMULATIONS GIVEN
THE BOUNDARY LAYER TEMPERATURES. FOR TIMING…PREFER A SOLUTION
CLOSER TO THE 12Z EC…THAT IS SLOWER…………
==============
Here is the full “area forecast discussion”:http://forecast.weather.gov/product.php?site=NWS&issuedby=LOT&product=AFD&format=CI&version=1&glossary=1
Most of it is way over my head, but when the models diverge from observations, the forecaster has no qualms about throwing out model solutions. Good stuff.

Professor Armstrong: Thank you for an informative, educational and excellent post!!
With the trillions of bucks sent for research; one would think that at least a smidgeon would’ve been spent on researching, designing and TESTING the models that are driving spending trillions.
I have difficulty understanding how scientists fail to design tests to prove a model works BEFORE using said model. In the climate foo foo science land apparently a program/model that finishes it’s run is considered darn good stuff to use.
Believers accept shoddy model results along with adhoc prediction modifications to account for model/prediction fails; they then seek to proselytize students by overwhelming the children with forced propaganda. Is a science student with an ‘A’ in climate doodling an honors science student?

In an ongoing study, we have identified 26 alarmist movements that were similar to the current man-made global warming alarm (e.g., population growth and famine in the 1960s, and global cooling in the 1970s). In all cases, human activity was predicted to cause environmental catastrophe and harm to people.
Despite strong support from leading scientists, none of the alarmist movements relied on scientific forecasting methods.
The government imposed regulations in 23 of the 25 alarms that involved calls for government intervention. None of the alarming forecasts turned out to be correct. Of the 23 cases involving government interventions, none were effective, and 20 caused net harm.

If you ever needed proof of your seer-sucker theory, there it is … and the government is always the biggest sucker …
w.

When I was a forecaster some 30 years ago, we only used numeric guidance (ie models) for “guidance”. You could always tell a newbie by his over-reliance on forecast models (in those days, it was the LFM and Spectral Models). There isn’t a GCM anywhere that has shown any kind of precision in forecasting future temperatures on a global scale. The models have to be constantly re-adjusted to reflect real data.

The inability by Prof. Armstrong and his colleagues to identify a single scientific forecast in the literature of global warming climatology follows from the failure on the part of climatologists to identify the events in the statistical population that underlies these forecasts.

The inability by Prof. Armstrong and his colleagues to identify a single scientific forecast in the literature of global warming climatology follows from the failure on the part of climatologists to identify the events in the statistical population that underlies these forecasts.

Perhaps, perhaps … but every single time I’ve asked you to explain what the hell you mean by the “statistical population”, you wander off into some of the most meaningless verbiage it has been my pleasure to see …
As a result, let me ask you NOT to try to explain it, you appear to be totally unqualified. If you have a friend that could give the explanation at try, that would be great, but you’ve had your three strikes and you’ve struck out.
w.

Willis Eschenbach:
In debating a scientific issue, the personalities of the debaters are irrelevant. However, as you have introduced my person as a topic in this thread I am forced to address it. It appears to me that the Willis Eschenbach doctrine of “three strikes and you’ve struck out” is an expression of Willis Eschenbach pig headedness. Perhaps it is an expression of something more sinister.
In my writings in this thread, the term “statistical population” references a collection of independent events.As my definition of this term is not unusual, I believe you should look to your own misunderstanding of the bases for mathematical statisttics as a possible explanation for your inability to understand what I say.
My guess is that a more likely source of trouble for you is a lack of background in the equivocation fallacy.in the context of global warming climatology. This fallacy is invoked whenever a person treats the terms “prediction” (aka forecast) and “projection” as synonyms. Predictions reference a statistical population but projections do not. In treating the two terms as synonyms, climatologists imply the existence of a statistical population when there isn’t one. One consequence from the lack of a statistical population is the non-falsifiability of the claims that are made by the climate models. Another is a lack of the information that could otherwise be supplied by the climate models to policy makers about the outcomes from their policy decisions. Exploitation of the exquivocation fallacy makes it seem as though policy makes have information when they have none.
In your hide-bound support for conflation of the term “prediction” with the term “projection” you’ve played into the hands of people who would like us to think policy makers have information when they have none. I’d like to turn this situation around. Are you on board or not?

Willis Eschenbach:
In debating a scientific issue, the personalities of the debaters are irrelevant. However, as you have introduced my person as a topic in this thread I am forced to address it. It appears to me that the Willis Eschenbach doctrine of “three strikes and you’ve struck out” is an expression of Willis Eschenbach pig headedness. Perhaps it is an expression of something more sinister.

Your person? I could care less about your person. What I care about is that several times I’ve asked you for an explanation of why a collection of say temperatures at five o’clock every day are NOT a sample from the statistical population of all 5:00 temperatures of the last hundred years. I have never gotten any understandable answer from you.
That’s the problem, not your “person”.

In my writings in this thread, the term “statistical population” references a collection of independent events.As my definition of this term is not unusual, I believe you should look to your own misunderstanding of the bases for mathematical statisttics as a possible explanation for your inability to understand what I say.

I find your definition of the term highly unusual. From the University of California:

Statistical Population
• The entire underlying set of observations from which samples are drawn.

Nothing in there about a “collection of independent events”. They say the sample is drawn from a set of observations they call the statistical population. You know, observations like the temperature observations you claim are NOT a statistical population … so their definition sure as hell doesn’t agree with yours.
On the other hand, the OECD says:

STATISTICAL POPULATION
Definition:
Population is the total membership or population or “universe” of a defined class of people, objects or events.
Context:
There are two types of population, viz, target population and survey population.
A target population is the population outlined in the survey objects about which information is to be sought and a survey population is the population from which information can be obtained in the survey.

Again, nothing like your definition … then from Princeton University:

A statistical population is a set of entities concerning which statistical inferences are to be drawn, often based on a random sample taken from the population.

Now, none of them indicate in there anywhere that a collection of temperature records taken at 5 PM every day is NOT a sample from a statistical population. The total population is all measurements at 5:00. The sample population are the actual observations.
But you say nooooo, that’s not a sample population … but just as has happened here again, you have NOT said why temperature observations are not a statistical population.
I KNOW what a statistical population is, my friend, and have for years. What I can’t figure out is why you think there’s no statistical population anywhere in climatology … and that I can’t get from your repeated and incorrect objection that you are just using the standard definition. You aren’t using it, which is why I keep asking.
As I have said all along, Terry, you could be right—I’ve just never gotten a coherent answer from you to see if you are right or not.
w.

Willis Eschenbach:
Rather than engage in a rather useless argument with you over the semantics of “statistical population,” I’ll disambiguate this term for use in making my argument, Disambiguation of the term will facilitate scrutiny of my argument for logical errors.
In probability theory, the idea of an “event” plays a leading role as the name of the set upon which probability is defined. Through the use of probability theory, one is able to create models of systems wherein information about the outcomes of events may be incomplete.
The empirical counterpart of a probability is a limiting relative frequency. The limiting relative frequency of a set of events of a particular description is the ratio of a pair of event counts or “frequencies.” That the associated events are countable implies their statistical independence.
The population that I have in mind is a set of independent events. The independence of these events is ensured by the fact that no two events overlap in time. Thus, each such event has starting time and an ending time which together define this event’s period of existence. That an event started at midnight Greenwich mean time on Jan. 1, 1900 and ended at midnight Greenwich mean time on Jan. 1, 1930 defines this event’s period of existence, for example.
Associated with my type of population is a predictive model. Associated with this model is a set of independent variables; the mean value of a specified CO2 time series in the period of a specified event is one possibility for an independent variable. An element in the Cartesian product of the values that are taken on by the various independent variables is an example of a tuple. By placement of the elements of a proper subset of the complete set of tuples in an inclusive disjunction, one forms an example of a “condition.” The proposition that “the mean value of the specified CO2 time series in the period of the specified event exceeds 400 ppm is an example of a condition. A condition is a state of (description of) an event.
Similarly, associated with the model is a set of dependent variables; the mean value of specified global temperature time series in the period of the specified event is one possibility for an independent variable An element in the Cartesian product of the values that are taken on by the various dependent variables is an example of a tuple. By placement of the elements in a proper subset of the tuples in the complete set of them in an inclusive disjunction, one forms an example of an “outcome.” The proposition that “the mean value of the specified global temperature time series in the period of the specified event exceeds 16 Celsius” is an example of an outcome An outcome is a state of (description of) an event.
At the time a prediction is made, the condition has been observed. At this time, the outcome has not been observed but it is susceptible to being inferred. When it occurs, the outcome is susceptible to being observed.
A subset of a population in which the condition and outcome have both been observed is an example of a “sample.” Each element of a sample is an example of an “observed event.” Thus, a sample is a collection of observed events.
In the circumstance that a model has been tested without being falsified in a sample not used in construction of the model, by comparison of the predicted probability values to the observed relative frequency values, this model is said to be “validated.” Long ago, the IPCC expert reviewer Vincent Gray complained to IPCC managment that the IPCC’s models were insusceptible to being validated. In the essay entitled “Spinning the Climate” (http://icecap.us/images/uploads/SPINNING_THE_CLIMATE08.pdf ) Gray describes the
IPCC’s response to his complaint. This was to establish a policy of substituting the
term “evaluate” for the similar sounding term “validate” and the term “projection”
for the similar sounding term “prediction.” In an IPCC-style “evaluation,” a global
temperature time series was plotted along side one or more projections. Comparisons
of predicted probabilities to observed relative frequencies could not be made, for neither probabilities nor relative frequencies could exist in lieu of the missing statistical
population. However, in the circumstance that there was but a single projection,
statistics such as the root mean squared error could be computed. It is this kind of
statistic that was used by Prof. Armstrong and his colleagues in reaching their
conclusions about the IPCC climate models.. Armstrong et al could not have compared predicted probabilities to observed relative frequencies as neither existed for these models.
A necessary component of a control system is a model that predicts the outcomes of
events with a degree of reliability by inferring these outcomes in the period before
they become available for observation. Despite the expenditure of several hundred
billion US$, global warming research has not yet produced such a component. It
follows that it is not currently possible for governments to control the climate via
curbs on CO2 emissions. Nonetheless, governments (including the governments of the U.S. and of California) are proceeding to try to do so.
Policy makers, politicians and climatologists are falling for the deception that conflates “prediction” with the similar sounding term “projection” and “validation” with the similar sounding term “evaluation” as facilitated by the IPCC’s response to Dr. Gray. For you to suggest that a temperature time series was an example of a “statistical population” as I’ve defined this term would be for you to be a party to this deception.
In logical terms, the deception being worked is an example of the equivocation
fallacy. An “equivocation” is an argument in which a polysemic term (term with more
than one meaning) changes meaning in the middle of the argument. A principle of logic
states that one cannot draw a proper inference from an equivocation. To draw an
improper inference from an equivocation is the equivocation fallacy. In AR4, the IPCC
makes extensive use of the equivocation fallacy in reaching its conclusions (http://judithcurry.com/2011/02/15/the-principles-of-reasoning-part-iii-logic-and-
climatology/ )..

Brahan the Seer was quite good at predicting floods but other climate forecasters should take heed from his ultimate demise when he was allegedly burnt in a spiked tar barrel, on the command of his Earl’s wife, Lady Seaforth

Hi, jorgekafkazar, Nye Bevan was a cultured man, and I think in this quote he’s using ‘book’ to refer metaphorically to knowledge, in contrast to guessing; but if he was inspired by one particular book, I’d guess that it was ‘The Ragged-Trousered Philanthropists’ by Robert Tressell. Well worth reading, especially for those in the US (I’m British) who think that standard, almost boring, European social democracy is radical communism.
Wikipedia summary: http://en.wikipedia.org/wiki/The_Ragged_Trousered_Philanthropists . Text in various formats (it’s out of copyright) http://www.gutenberg.org/ebooks/3608 .
The British labour movement has not, by and large, been Marxist (the membership of the British Communist Party was pitifully small compared with that of the French and Italian Parties, even though it was a pioneer of Eurocommunism, the posture that CPs did not have to be revolutionary but could work within the parliamentary democratic system, and the Trots have been a minor irritation).
I want to respect the preference of WUWT for not getting into extended political debate, but if you want to take this up, please suggest a forum.
(By the way, there’s no harm in reading Das Kapital and other works by Marx and Engels; they’re an important part of 19th century economic and social theory, and can still contribute insights, even though the over-arching political and economic theory has been roundly disproved in practice.)

We use cookies to ensure that we give you the best experience on WUWT. If you continue to use this site we will assume that you are happy with it. This notice is required by recently enacted EU GDPR rules, and since WUWT is a globally read website, we need to keep the bureaucrats off our case!OkPrivacy policy