All posts tagged Max Fisher

Last week, Vox ran a long piece by Max Fisher claiming that “the prospect of a major war, even a nuclear war, in Europe has become thinkable, [experts] warn, even plausible.” Without ever clarifying what “thinkable” or “plausible” mean in this context, Fisher seems to be arguing that, while still unlikely, the probability of a nuclear war between the United States and Russia is no longer small and is rising.

I finished Fisher’s piece and wondered: Is that true? As someone who’s worked on a couple of projects (here and here) that use “wisdom of crowds” methods to make educated guesses about how likely various geopolitical events are, I know that one way to try to answer that question is to ask a bunch of informed people for their best estimates and then average them.

So, on Thursday morning, I went to SurveyMonkey and set up a two-question survey that asks respondents to assess the likelihood of war between the United States and Russia before 2020 and, if war were to happen, the likelihood that one or both sides would use nuclear weapons. To elicit responses, I tweeted the link once and posted it to the Conflict Research Group on Facebook and the IRstudies subreddit. The survey is still running [UPDATE: It’s now closed, because Survey Monkey won’t show me more than the first 100 responses without a paid subscription], but 100 people have taken it so far, and here are the results—first, on the risk of war:

And then on the risk that one or both sides would nuclear weapons, conditional on the occurrence of war:

These results come from a convenience sample, so we shouldn’t put too much stock in them. Still, my confidence in their reliability got a boost when I learned yesterday that a recent survey of international-relations experts around the world asked an almost-identical question about the risk of a war and obtained similar results. In its 2014 survey, the TRIP project asked: “How likely is war between the United States and Russia over the next decade? Please use the 0–10 scale with 10 indicating that war will definitely occur.” They got 2,040 valid responses to that question, and here’s how they were distributed:

Those results are centered a little further to the right than the ones from my survey, but TRIP asked about a longer time period (“next decade” vs. “before 2020″), and those additional five years could explain the difference. It’s also important to note that the scales aren’t directly comparable; where the TRIP survey’s bins implicitly lie on a linear scale, mine were labeled to give respondents more options toward the extremes (e.g., “Certainly not” and “Almost certainly not”).

In light of that corroborating evidence, let’s assume for the moment that the responses to my survey are not junk. So then, how likely is a US/Russia war in the next several years, and how likely is it that such a war would go nuclear if it happened? To get to estimated probabilities of those events, I did two things:

Here are the estimates that process produced, rounded up to the nearest whole percentage point:

Probability of war: 11%

Probability that one or both sides will use nuclear weapons, conditional on war: 18%

To translate those figures into a single number representing the crowd’s estimate of the probability of nuclear war between the US and Russia before 2020, we take their product: 2%.

Is that number different from what Max Fisher had in mind when he wrote that a nuclear war between the US and Russia is now “thinkable,” “plausible,” and “more likely than you think”? I don’t know. To me, “thinkable” and “plausible” seem about as specific as “possible,” a descriptor that applies to almost any geopolitical event you can imagine. I think Max’s chief concern in writing that piece was to draw attention to a risk that he believes to be dangerously under-appreciated, but it would be nice if he had asked his sources to be more specific about just how likely they think this calamity is.

More important, is that estimate “true”? As Ralph Atkins argued in a recent Financial Times piece about estimating the odds of Grexit, it’s impossible to say. For unprecedented and at least partially unique events like these—an exit from the euro zone, or a nuclear war between major powers—we can never know the event-generating process well enough to estimate their probabilities with high confidence. What we get instead are summaries of peoples’ current beliefs about those events’ likelihood. That’s highly imperfect, but it’s still informative in its own way.

The post is great because it succeeds in condensing a large and complex literature into a small set of findings directly relevant to an important topic of public concern. It’s no coincidence that this post was written by one of the leading scholars on that subject. A “data scientist” could have looked at the same data sets used in the studies on which Walter bases her summary and not known which statistics would be most informative. Even with the right statistics in hand, a “hacker” probably wouldn’t know much about the relative quality of the different data sources, or the comparative-historical evidence on relevant causal mechanisms—two things that could (and should) inform their thinking about how much confidence to attach to the various results. To me, this is a nice illustration of the point that, even in an era of relentless quantification, subject-matter expertise still matters.

The one thing that seems to have gotten lost in the retellings and retweetings of this distilled evidence, though, is the idea of uncertainty. Apparently inspired by Walter’s post, Max Fisher wrote a similar one for the Washington Post‘s Worldviews blog under the headline “Political science says Syria’s civil war will probably last at least another decade.” Fisher’s prose is appropriately less specific than that (erroneous) headline, but if my Twitter feed is any indication, lots of people read Walter’s and Fisher’s post as predictions that the Syrian war will probably last 10 years or more in total.*

If you had to bet now on the war’s eventual duration, you’d be right to expect an over-under around 10, but the smart play would probably be not to bet at all, unless you were offered very favorable odds or you had some solid hedges in place. That’s because the statistics Walter and Fisher cite are based on a relatively small number of instances of a complex phenomenon, the origins and dynamics of which we still poorly understand. Under these circumstances, statistical forecasting is inevitably imprecise, and the imprecision only increases the farther we try to peer into the future.

We can visualize that imprecision, and the uncertainty it represents, with something called a prediction interval. A prediction interval is just an estimate of the range in which we expect future values of our quantity of interest to fall with some probability. Prediction intervals are sometimes included in plots of time-series forecasts, and the results typically look like the bell of a trumpet, as shown in the example below. The farther into the future you try to look, the less confidence you should have in your point prediction. When working with noisy data on a stochastic process, it doesn’t take a lot of time slices to reach the point where your prediction interval practically spans the full range of possible values.

Civil wars are, without question, one of those stochastic processes with noisy data. The averages Walter and Fisher cite are just central tendencies from a pretty heterogenous set of cases observed over a long period of world history. Using data like these, I think we can be very confident that the war will last at least a few more months and somewhat confident that it will last at least another year or more. Beyond that, though, I’d say the bell of our forecasting trumpet widens very quickly, and I wouldn’t want to hazard a guess if I didn’t have to.

* In fact, neither Walter nor Fisher specifically predicted that the war would last x number of years or more. Here’s what Walter actually wrote:

I think that’s a nice verbal summary of the statistical uncertainty I’m trying to underscore. And here’s what Fisher wrote under that misleading headline:

According to studies of intra-state conflicts since 1945, civil wars tend to last an average of about seven to 12 years. That would put the end of the war somewhere between 2018 and 2023.

Worse, those studies have identified several factors that tend to make civil wars last even longer than the average. A number of those factors appear to apply to Syria, suggesting that this war could be an unusually long one. Of course, those are just estimates based on averages; by definition, half of all civil wars are shorter than the median length, and Syria’s could be one of them. But, based on the political science, Syria has the right conditions to last through President Obama’s tenure and perhaps most or all of his successor’s.

I’m grateful for the attention Max has drawn to my work, but if it had been up to me, I would have done the mapping a little differently. As I said to Max in an email from which he later excerpted, the forecasts simply aren’t sharp enough to parse the world as finely as their map did. Our theories of what causes coup attempts are too fuzzy and our measures of the things in those theories are too spotty to estimate the probability of these rare events with that much precision.

But, hey, I’m a data guy. I don’t have to stick to grumbling about the Post‘s map; I can make my own! So…

The map below sorts the countries of the world into three groups based on their relative coup risk for 2013: highest (red), moderate (orange), and lowest (beige). I emphasize “relative” because coup attempts are very rare, so the estimated risk of coup attempts in any given country in any single year is pretty small. For example, Guinea-Bissau tops my list for 2013, and the estimated probability of at least one coup attempt occurring there this year is only 25%. Most countries worldwide are under 2%.

Consistent with an emphasis on relative risk, the categories I’ve mapped are based on rank order, not predicted probability. The riskiest fifth of the world (33 countries) makes up the “highest” group, the second fifth the “moderate” group, and the bottom three-fifths the “lowest” group.

This forecasting process doesn’t have enough of track record for me to say exactly how those categories relate to real-world risk, but based on my experience working with similar data and models, I would expect roughly four of every five coup attempts to occur in countries identified here as high risk, and the occasional “miss” to come from the moderate-risk set. Only very rarely should coup attempts come from the 100 or so countries in the low-risk group.