Armed conflict study's results don't add up for two main reasons

The claim came from a press release for the 2017 edition of the Armed Conflict Survey (ACS) by the International Institute for Strategic Studies (IISS), a London-based think tank. And a media frenzy followed.

The IISS press release stated that the 23,000 deaths in 2016 that it attributes to Mexico’s struggle against organized crime came second to the 50,000 fatalities caused by Syria’s civil war.

They said this was higher than those caused by conflicts in the rest of the countries covered by the survey, including Iraq, Afghanistan, Yemen and the so-called “Northern Triangle” of El Salvador, Honduras and Guatemala.

The claim was shared extensively, including via a retweet by U.S. President Donald Trump. But it drew swift criticism from Mexico’s government, which argued, among other things, that Mexico’s organized crime-related violence is not comparable to armed conflicts in Syria or Iraq.

In response, IISS published a blog post outlining why it thinks Mexico is in a state of conflict, including Mexico’s own characterization of criminal groups as an existential threat under the previous government.

But I argue that the comparison placing Mexico as the second-deadliest conflict zone is fallacious for two main reasons.

First, the comparison did not take into account the vastly different sizes of the countries in the study. Mexico’s population (127 million) is larger than the most recent population figures for Syria, Iraq, Afghanistan and Yemen combined (114.2 million).

All else being equal, it is expected that countries with larger populations will have more fatalities or homicides in absolute terms. For example, there where almost twice as many homicides in the U.S. (12,000) as there were fatalities in Yemen (7,000), but this is because the U.S. has a population that is 12 times bigger than Yemen’s, not because it is deadlier than Yemen.

To account for different population sizes in cross-national comparisons, mortality and homicide figures are usually expressed as incidents per 100,000 people. This captures the probability that individuals in a given country face of being killed, and so are a better measure than absolute figures.

As seen in the graph below, once population sizes are taken into account, the probability of being killed in Mexico was much lower than in the rest of the countries in the ACS top five or in the Northern Triangle.

Second, the figures being compared are not measuring the same phenomenon. In the report, fatalities in Mexico and Central America refer to the total number of intentional homicides, while in the rest of the countries in the study they refer to deaths directly caused by armed conflicts.

It is true that intentional homicides are sometimes used as a proxy measurement of organized crime activity in Mexico – something I am exploring in my ongoing research. However, that does not mean that it is appropriate to uncritically compare the absolute number of homicides in Mexico to conflict-related fatalities in other countries.

That would suggest that all intentional homicides in Mexico – high as they may be – can be attributed to organized crime violence, which is far from the case.

Previous editions of the ACS in 2015 and 2016 also included Mexico and reported similarly high levels of fatalities. But the level of coverage of the 2017 report reflects a vastly different global public agenda.

Today, with Mexico-U.S. relations at a new low, a looming renegotiation of the North American Free Trade Agreement and the promise of a new physical wall between the two countries, it is not surprising that IISS’s press statement landed front and center in the international news media – even if such comparisons are not the main focus of the ACS.

Journalists undeniably “have to get better at reporting science.” But academics and those involved in research – especially in controversial issues such as crime and security – also need to exercise care when publicizing their findings.

There is a tension between the efforts to publicize research by highlighting headline-grabbing findings, and the risk of having such findings overhyped and misrepresented, or even spun into a deliberate political agenda, as the right-wing Breitbart News did with the IISS announcement.

Academics and think tanks may not have much control over how their findings are reported by the media once they are released to the public. So it is imperative to take extra care when communicating them to the press, and setting out clearly what are the limitations of the findings, as well as the generalizations that can accurately be drawn from them.