Why you should be wary of statistics on ‘modern slavery’ and ‘trafficking’

In this Nov. 29, 2014, image from video, a former slave from Burma who goes by the name Mozet, center, one of several slaves who escaped or ran away while Thai trawlers were docked at the Benjina port, cuts planks from a tree to earn money for food. Because the men were brought to Indonesia illegally — many after being tricked, sold or kidnapped by Thai brokers — they do not have any official documents and live in constant fear of being arrested. (AP)

“This report estimates that, based on the information governments have provided, only around 40,000 victims have been identified in the last year. In contrast, social scientists estimate that as many as 27 million men, women, and children are trafficking victims at any given time.”

— Introduction to the State Department’s Trafficking in Persons report, June 2013

“Our work with victims is the key that will open the door to real change — not just on behalf of the more than 44,000 survivors who have been identified in the past year, but also for the more than 20 million victims of trafficking who have not.”

— Introduction to the State Department’s Trafficking in Persons report, June 2014

Imagine that. In the space of one year, the number of victims of trafficking declined by 7 million in an official U.S. government report.

But not to worry. There’s something called the Global Slavery Index (GSI), which received fawning publicity, including in The Washington Post. In 2013, the GSI, sponsored by the Walk Free Foundation, estimated that there were 29.8 million people in “modern slavery” around the world. In November 2014, the GSI unveiled what it described as a more precise estimate: 35.8 million people.

That’s an increase of 6 million people! What’s going on here?

The Facts

Human trafficking — or, as some prefer, “modern slavery” — is a largely hidden crime, so data are relatively scarce. Note that in the State Department reports, there is a large gulf between the estimates of tens of millions of victims and the actual number of identified “survivors” — 44,000 at last count. (This number is also a bit dubious.)

Moreover, the numbers can vary dramatically depending on the definition — and increasingly, the definition has been stretched. A U.N. protocol on trafficking, adopted in 2000, provided a definition of trafficking that for political reasons was kept deliberately vague: Trafficking must meet three conditions — an act (such as movement), means (coercion) and purpose (exploitation). Then, in what American University law professor Janie A. Chuang calls “exploitation creep,” trafficking over time has been recast to include all forced labor, even if a person does not change location, and then has been relabeled as “modern slavery.”

When the State Department set up its office on trafficking in the early 2000s, the numbers were much more modest. The Department’s 2002 report provided an estimate that “at least 700,000, and possibly as many as four million men, women and children worldwide were bought, sold, transported and held against their will in slave-like conditions.” At the time, the George W. Bush administration was largely focused on highlighting anti-prostitution efforts.

The Obama administration broadened the focus on trafficking to include forced labor, including when no movement was involved — and officials began to label all trafficking as “modern slavery.”

A State Department official, who asked not to be identified, said that the 27 million figure used in the 2013 report came from an estimate by Kevin Bales, a professor at Britain’s University of Hull and author of a 2007 book, “Ending Slavery.”

Then, for the 2014 report, State Department officials decided to rely on a 20.9 million estimate issued in 2012 by the International Labor Organization because officials decided it was more reputable. (This figure was a huge increase from a 2005 ILO estimate of 12.3 million.)

In an example of how definitions matter, 9.1 million of the estimated victims in the ILO report were moved internally or internationally. “The majority, 11.8 million (56 percent), are subjected to forced labor in their place of origin or residence,” the report said.

“The major problem we have always faced with human trafficking is finding good data,” the State Department official said. “For now, this is still a guesstimate, but the best guesstimate there is.” The official added: “I noticed that media likes to cite the Global Slavery Index number of 35.8 million because it’s much larger.”

This brings us to the Global Slavery Index. Bales no longer stands by his estimate of 27 million, saying it dates from the 1990s, and points to the GSI as more accurate. (He is the lead author.) But the GSI figure has come under attack from other researchers for having a murky, inconsistent and questionable methodology.

The Walk Free Foundation, founded in 2012 by Australian billionaire Andrew “Twiggy” Forrest, says it wants to eliminate slavery in a generation. The GSI not only provides a total but purports to show how many “slaves” are in each country. GSI relies on an expansive definition of slavery, but confusingly it relies on primary and secondary data that was collected under different definitions.

The data are relatively sparse, but the GSI extrapolates from existing numbers to make calculations in what it deems are similar countries. Essentially, researchers extrapolated from 19 countries to come up with precise statistics for the 167 countries that make up the index.

Thus data for the United States is considered relevant to calculate Italy’s total of 11,400 slaves, for instance. South Africa’s number of slaves — supposedly 106,000 — was derived from the fact that GSI researchers decided the country is 70 percent “Western Europe” and 30 percent “African” (specifically, an amalgam of Ethiopia, Nigeria, Niger and Namibia).

In the most recent report, GSI began to introduce Gallup polling in selected countries. This is one reason it says the number jumped by 6 million in one year: “The increase is due to the improved accuracy and precision of our measures and that we are uncovering modern slavery where it was not seen before.”

The polling was done face to face in seven countries, and 19 additional polls will be added to next year’s index, said Pablo Diego Rosell, a Gallup consultant. “In applicable countries, Gallup uses a network sampling methodology. Network sampling gathers information about an individual’s carefully defined family network, including those who may be living elsewhere,” he said. “This approach is most efficient in the countries that Walk Free prioritized for survey data collection, where modern slavery is either highly prevalent or has greater visibility.” In other countries, where he said slavery is not as visible, the index uses “non-survey methodologies.”

But Andrew Guth, who wrote critically of GSI’s methodology, notes that the Gallup polling, if taken at face value, demonstrated that some of the index’s previous estimates were off course. Ethiopia turned out to have a prevalence level five times lower than the year before — while Russia was deemed to be two times higher.

“This simple comparison throws into question the reliability of their estimates and extrapolations of other countries, whether from last year or this year,” he said. “Even their own data compared to itself is not reliable.”

It is beyond the scope of this column to assess the merits of this debate, which largely is between experts such as Ronald Weitzer of George Washington University, who advocate for careful studies of local problems, and those such as Bales, who press for macro-level estimates.

“The challenge is that modern slavery is a hidden crime. Every country has made slavery illegal, so collecting data on this matter has not been easy,” said Sheldon Zhang of San Diego State University, who consults for Walk Free. “We should not abandon macro-level estimation just because it is full of problems.”

Weitzer says it is more than a philosophical debate but one with important consequences. “It matters a great deal in terms of (1) whether human trafficking or modern slavery is indeed a huge problem and (2) all the money spent fighting the problem, and the proliferation of more and more laws and creation of police anti-trafficking units,” he said. “The bottom line would be: what is the source of the figures propounded by NGOs, governments, international organizations and some scholars.”

The Pinocchio Test

Clearly there is a problem with the numbers when the U.S. government cites a figure of 20 million and a well-funded, media-savvy organization touts a figure of “slaves” that is almost twice as high.

Media organizations are complicit in fostering misperceptions by often citing these figures as established fact, without even an explanation or examination of the methodology. The numbers grow or shrink depending on the definitions that are used, and yet media reports rarely examine that aspect. (A rare exception is a 2007 article by our colleague Jerry Markon, who documented how few actual victims of human trafficking have been found. More recently, the Guardian has published articles critical of GSI.)

Advocates want to call attention to a serious problem, and big numbers of course attract media attention. But these guesstimates remain too shaky to be cited without a healthy dose of skepticism. The estimates may be done in good faith, so these Pinocchios are for all-too-credulous acceptance of them.