Political Scientists Are Lousy Forecasters

DESPERATE “Action Alerts” land in my in-box. They’re from the American Political Science Association and colleagues, many of whom fear grave “threats” to our discipline. As a defense, they’ve supplied “talking points” we can use to tell Congressional representatives that political science is a “critical part of our national science agenda.”

Political scientists are defensive these days because in May the House passed an amendment to a bill eliminating National Science Foundation grants for political scientists. Soon the Senate may vote on similar legislation. Colleagues, especially those who have received N.S.F. grants, will loathe me for saying this, but just this once I’m sympathetic with the anti-intellectual Republicans behind this amendment. Why? The bill incited a national conversation about a subject that has troubled me for decades: the government — disproportionately — supports research that is amenable to statistical analyses and models even though everyone knows the clean equations mask messy realities that contrived data sets and assumptions don’t, and can’t, capture.

It’s an open secret in my discipline: in terms of accurate political predictions (the field’s benchmark for what counts as science), my colleagues have failed spectacularly and wasted colossal amounts of time and money. The most obvious example may be political scientists’ insistence, during the cold war, that the Soviet Union would persist as a nuclear threat to the United States. In 1993, in the journal International Security, for example, the cold war historian John Lewis Gaddis wrote that the demise of the Soviet Union was “of such importance that no approach to the study of international relations claiming both foresight and competence should have failed to see it coming.” And yet, he noted, “None actually did so.” Careers were made, prizes awarded and millions of research dollars distributed to international relations experts, even though Nancy Reagan’s astrologer may have had superior forecasting skills.

Political prognosticators fare just as poorly on domestic politics. In a peer-reviewed journal, the political scientist Morris P. Fiorina wrote that “we seem to have settled into a persistent pattern of divided government” — of Republican presidents and Democratic Congresses. Professor Fiorina’s ideas, which synced nicely with the conventional wisdom at the time, appeared in an article in 1992 — just before the Democrat Bill Clinton’s presidential victory and the Republican 1994 takeover of the House.

Alas, little has changed. Did any prominent N.S.F.-financed researchers predict that an organization like Al Qaeda would change global and domestic politics for at least a generation? Nope. Or that the Arab Spring would overthrow leaders in Egypt, Libya and Tunisia? No, again. What about proposals for research into questions that might favor Democratic politics and that political scientists seeking N.S.F. financing do not ask — perhaps, one colleague suggests, because N.S.F. program officers discourage them? Why are my colleagues kowtowing to Congress for research money that comes with ideological strings attached?

The political scientist Ted Hopf wrote in a 1993 article that experts failed to anticipate the Soviet Union’s collapse largely because the military establishment played such a big role in setting the government’s financing priorities. “Directed by this logic of the cold war, research dollars flowed from private foundations, government agencies and military individual bureaucracies.” Now, nearly 20 years later, the A.P.S.A. Web site trumpets my colleagues’ collaboration with the government, “most notably in the area of defense,” as a reason to retain political science N.S.F. financing.

Many of today’s peer-reviewed studies offer trivial confirmations of the obvious and policy documents filled with egregious, dangerous errors. My colleagues now point to research by the political scientists and N.S.F. grant recipients James D. Fearon and David D. Laitin that claims that civil wars result from weak states, and are not caused by ethnic grievances. Numerous scholars have, however, convincingly criticized Professors Fearon and Laitin’s work. In 2011 Lars-Erik Cederman, Nils B. Weidmann and Kristian Skrede Gleditsch wrote in the American Political Science Review that “rejecting ‘messy’ factors, like grievances and inequalities,” which are hard to quantify, “may lead to more elegant models that can be more easily tested, but the fact remains that some of the most intractable and damaging conflict processes in the contemporary world, including Sudan and the former Yugoslavia, are largely about political and economic injustice,” an observation that policy makers could glean from a subscription to this newspaper and that nonetheless is more astute than the insights offered by Professors Fearon and Laitin.

An error has occurred. Please try again later.

You are already subscribed to this email.

How do we know that these examples aren’t atypical cherries picked by a political theorist munching sour grapes? Because in the 1980s, the political psychologist Philip E. Tetlock began systematically quizzing 284 political experts — most of whom were political science Ph.D.’s — on dozens of basic questions, like whether a country would go to war, leave NATO or change its boundaries or a political leader would remain in office. His book “Expert Political Judgment: How Good Is It? How Can We Know?” won the A.P.S.A.’s prize for the best book published on government, politics or international affairs.

Professor Tetlock’s main finding? Chimps randomly throwing darts at the possible outcomes would have done almost as well as the experts.

These results wouldn’t surprise the guru of the scientific method, Karl Popper, whose 1934 book “The Logic of Scientific Discovery” remains the cornerstone of the scientific method. Yet Mr. Popper himself scoffed at the pretensions of the social sciences: “Long-term prophecies can be derived from scientific conditional predictions only if they apply to systems which can be described as well-isolated, stationary, and recurrent. These systems are very rare in nature; and modern society is not one of them.”

Government can — and should — assist political scientists, especially those who use history and theory to explain shifting political contexts, challenge our intuitions and help us see beyond daily newspaper headlines. Research aimed at political prediction is doomed to fail. At least if the idea is to predict more accurately than a dart-throwing chimp.

To shield research from disciplinary biases of the moment, the government should finance scholars through a lottery: anyone with a political science Ph.D. and a defensible budget could apply for grants at different financing levels. And of course government needs to finance graduate student studies and thorough demographic, political and economic data collection. I look forward to seeing what happens to my discipline and politics more generally once we stop mistaking probability studies and statistical significance for knowledge.

Jacqueline Stevens is a professor of political science at Northwestern University and the author, most recently, of “States Without Nations: Citizenship for Mortals.”

A version of this op-ed appears in print on June 24, 2012, on Page SR6 of the New York edition with the headline: Political Scientists Are Lousy Forecasters. Today's Paper|Subscribe