Executive Summary

Prediction In Policy: A Process, Not A Product

By Daniel Sarewitz and Radford Byerly, Jr.

The Earth sciences, backed by formidable arrays of data gathering and
processing technologies, now offers the apparently credible promise of
predicting the future of nature; policy, under pressure as always to
deliver public benefit at low cost and even lower risk, has strong
incentives to accept this promise as one response to environmental issues.

We are investigating the role of prediction in the making of environmental
policies. Such policies relate to problems that include: planning for and
responding to natural hazards (weather, floods, earthquakes, asteroids);
planning for and responding to anthropogenic hazards (global climate
change, acid rain, nuclear waste); managing natural resources (oil
reserves, beaches); and regulating environmental impacts (mining).

On September 10-12, 1998, we (along with our co-investigators, Roger
Pielke, Jr., and Dale Jamieson) convened a workshop in Estes Park,
Colorado, that brought together a diverse group of people who are involved
in various ways with the process of prediction. Among the 35 participants
were: a scientist who works on climate models; the former emergency
manager of a major California city; a banker from a coastal city that is
subject to hurricanes; a seismologist; a rancher; a former official at the
federal Office of Management and Budget; an engineer who works on nuclear
waste isolation; a coastal geologist who studies beach erosion. The goal
of the workshop was to apply the collective wisdom of a range of
stakeholders (including natural scientists who make predictions, and social
scientists concerned with their use) to the problem of how scientific
predictions should be used (or not used) in the development of effective
policies relating to natural hazards, natural resources, and the
environment.

Consider two situations:

I. People listen to or read the morning weather report, and then make a
decision about clothing, accessories (umbrella? gloves? hat?), mode of
transport. This decision is backed by a personal experience of weather and
its local fluctuations, and a scientific and technical support
infrastructure that in the U.S. issues on the order of 10 million weather
predictions per year. These predictions, based in part on real-time
observations of weather patterns, are aimed at supporting the specific
information that people need. The consequences of a poor prediction, or a
poor decision based on a good prediction, are often modest-a wet shirt,
perhaps a car skidding off the road-although on rare occasions severe-an
airplane crash, the failure to evacuate a town. In either case, users have
accumulated enough experience in comparing the prediction to the actual
event, to develop a comfort with-to personally "calibrate"-the weather
prediction process.

II. In other situations, such personal experience is not possible.
Members of Congress listen to testimony from scientists about nuclear waste
disposal. Because radioactive waste remains dangerous for hundreds of
thousands of years, disposal systems must operate effectively for at least
that long. The relevant science uses analogy, mathematical models, and
extrapolation to predict events far in the future. Thus, there is no basis
in personal experience for evaluating or calibrating the actual performance
of the disposal systems or the science. Decisions must be based on
abstractions. Action must be taken, but the consequences of a poor
prediction, or a poor decision based on a good prediction, are potentially
disastrous, both politically (a lost election) and societally
(radionuclides leaking into groundwater or even reaching the atmosphere.)

Decision making is forward looking, so the allure of prediction is strong.
We look to predictions to help us make decisions that can mitigate or evade
the impact of nature on society, and of society on nature. In doing so,
we need to recognize that prediction has become part of a complex
decision-making process, a network of interrelationships that must function
well across all of its connections if predictions are to successfully serve
society. This integrated process involves policy makers (who solicit and
pay for predictions), scientists (who make predictions) and decision makers
(who use them--for everything from deciding whether to carry an umbrella to
evacuating a city in the path of a hurricane; for establishing levels of
insurance risk to negotiating an international environmental agreement).

The less frequent, less observable to the human eye, less spatially
discrete, more gradual, more distant in the future, and more severe a
predicted phenomenon, the more difficult it is to accumulate direct
experience. Where direct experience is sparse or lacking, other sources of
societal understanding must be developed, or the prediction process will
not function effectively. Science alone does not create this understanding.
What is necessary above all is an institutional structure that allows
policy makers, decision makers, and scientists to interact closely
throughout the entire prediction process, so that each knows the needs and
capabilities of the others. It is crucial that this process be open,
participatory, conducive to mutual respect. Efforts to shield expert
research and decision making from public scrutiny and accountability
invariably backfire and fuel distrust and counterproductive policies and
decisions.

How can the prediction process foster sound decision-making?

Predictions must be generated primarily with the needs of the user in
mind. Television weather predictions focus primarily on temperature,
precipitation, and wind, rather than thermal gradients, behavior of
aerosols, and barometric pressure. For scientists to participate usefully
in the prediction process, they must address the goals of the process, not
the goals of science; they must listen to stakeholders. For stakeholders
to participate usefully in this process, they must work closely and
persistently with the scientists to communicate their needs and problems.

The prediction process must be open. To create openness, stakeholders
must question predictions. For this questioning to be effective,
predictions should be as transparent as possible to the user. In
particular, assumptions, model limitations, and weaknesses in input data
should be forthrightly discussed. Especially in cases where personal
experience may be limited (acid rain, asteroid impacts, global warming),
public confidence in the validity of the prediction will derive in part
from an understanding of how the prediction is generated. Black boxes
generate distrust, especially when a prediction can stimulate decisions
that create winners and losers.

Even so, many types of predictions will never be understood by decision
makers in the way that weather predictions are understood. Experience is
important and cannot be replaced, but the prediction process can be
facilitated in other ways, for example, by being totally open about
predictions, warts and all; and by fully considering alternative approaches
to prediction, such as "no regrets" policies, adaptation, and better
planning and engineering.

Uncertainties must be clearly articulated (and understood) by the
scientists, so that users understand their implications. Failure to
understand uncertainties has contributed to poor decisions that then
undermine relations among scientists and decision makers; we saw this
during the Red River flood in Grand Forks, ND. But understanding the
uncertainties does not mean that the predictions will be useful. If policy
makers truly understood the uncertainties associated with predictions of
global climate change or nuclear waste behavior, they might decide that
strategies for action should not depend on predictions.

Alternatives to prediction must be evaluated as a part of the
prediction process. Rather than trying to predict the impacts of hard-rock
pit mines on water quality as a basis for environmental regulation, it
might be more feasible to spread risk through bonding or other types of
insurance. Predicting the consequences of global climate change has caused
policy gridlock; other approaches to mitigation and adaptation should be
more vigorously sought.

Predictions themselves must be viewed as events. The prediction
process must include mechanisms for the various stakeholders to fully
consider and plan what to do after a prediction is made.

When the prediction process is fostered by effective, participatory
institutions, and when a healthy decision environment emerges from these
institutions, then the products of predictive science may even become less
important. Earthquake prediction was once a policy priority; now it is
considered technically unfeasible, at least in the near future. But, in
California, the close-institutionalized-communication among scientists,
engineers, state and local officials, and the private sector, has led to
considerable advances in earthquake preparedness and a much decreased
dependence on prediction. On the other hand, in the absence of an
integrated and open decision environment, the scientific merit of
predictions can be rendered politically irrelevant, as has been seen with
nuclear waste disposal and acid rain. That is, if there is no adequate
decision environment for dealing with an event or situation, a
scientifically successful prediction may be no more useful than an
unsuccessful one.

These observations fly in the face of much current practice, where,
typically, policy makers recognize a problem, scientists then do research
to predict natural behavior associated with the problem, and predictions
are finally delivered to decision makers in the expectation that they will
be both useful and well-used. This sequence, which puts predictive
research at the core of the decision environment, rarely functions well in
practice. In contrast, our work suggests that, for virtually every
environmental problem, the key to effective decision making lies in
improving the decision environment itself. Such improvement may come from
cost-effective, politically realistic alternatives to prediction. The goal
of the decision environment must be good decisions, not good predictions.