This Is What Happened When Instagram Banned Certain Pro-Anorexia Words

In 2012, Instagram started moderating certain terms used by pro-anorexia groups, making it one of the more tightly moderated social media platforms. A group of Georgia Tech researchers decided to study if banning such words helped the communities using them. Instead, they found that it may make matters worse.
Like many social media sites, the photo and video sharing service found itself host to a number of communities that gave both users and owners pause. Groups of people with eating disorders formed communities to promote anorexia, bulimia and other eating disorders as a lifestyle. They traded tips on how to diet or purge, pictures of their bodies that made them feel ashamed and pushed them to diet more, and "thinspiration" photos of extremely thin celebrities.

So Instagram started aggressively moderating key terms like "thighgap", "imugly" and "thinspiration". Searching the terms produced no results, and when posts were tagged with those results, users had to first view a screen advising them that they would be viewing questionable content, and occasionally providing links to helpful sites or hotlines.

They found that banning exact terms did little to stop the communities who used the terms. Instead it spawned variations in spelling, such as "thyghgapp" and "thinspoooooo". Instagram banned 17 terms. Soon there were at least 250 terms specifically used by pro-eating disorder communities. The alternate spellings also popped up on tumblr and other social media platforms.

Did banning the terms foster an "us versus them" attitude in the communities, or make them double down on their commitment?

Graduate student Stevie Chancellor, one of the study's co-authors, told Gizmodo that she is reluctant to guess at the psychology of the people in the communities. "That's a correlation or causation problem," she said. But after the bans, the communities tended to talk about, and tag, more extreme behaviour. "Our observations show that people who used the new terms also used terms like 'isolation', 'loneliness' or 'self-harm'," she said. What caused the change isn't exactly clear. Chancellor believes it could be "not having the outside influence of someone saying 'come talk to me if you need me', instead of 'hey, good job, keep it up.'"

Instead of banning terms, De Choudhury and Chancellor think social media platforms should modify the recommendation system to show users "content that is very different from what they are looking for." A search for "thinspo" should return not just results for thinspiration photos, but for communities of people in recovery from eating disorders, body acceptance posts, social services posts and other posts that allow searchers to get a glimpse of other points of view and groups that might help them get healthier.

Might these insights be applicable to hate groups or criminal groups? They can certainly be just as inventive when it comes to dodging banned search terms. "It is human nature to come up with a different way out," said De Choudhury. However, the psychology and social context of hate groups is very different from those of people suffering from mental health problems or eating disorders, so in such cases, "it might make sense to ban things".

Chancellor agreed that "people are going to evade" restrictions on search terms under almost any circumstances. However, she thinks it might be possible for a similar modified approach to work in other groups: modifying search terms so that they return results that give people searching for them a different perspective or a wider base of knowledge. "This might be the first step in that general direction."

Trending Stories Right Now

It's becoming increasingly clear that Facebook has never faced a scandal like the one it's currently fighting through. Revelations over the weekend about its reckless sharing of user data sent its stock price plunging on Monday, and fresh calls for regulations on the social media network are looking more real than ever.

Incoming South Australian Premier Steven Marshall has today revealed that the Liberal government will not continue with Jay Weatherill's plan to install batteries in thousands of low-income households. The new government's plan will instead focus on means-tested subsidies for battery systems, and on the grid scale a focus on interconnectivity with NSW.