“Automatically identifying abuse is surprisingly difficult,” says Alex Krasodomski-Jones, who tracks online abuse as a researcher with the U.K.-based Centre for Analysis of Social Media. “The language of abuse is amorphous—changing frequently and often used in ways that do not connote abuse, such as when racially or sexually charged terms are appropriated by the groups they once denigrated. Given 10 tweets, a group of humans will rarely all agree on which ones should be classed as abusive, so you can imagine how difficult it would be for a computer.”

Detecting abusive language is often more difficult than one expects for a variety of reasons. the noisiness of the data in conjunction with a need for world knowledge not only makes this a challenging task to automate but also potentially a difficult task for people as well.

More than simple keyword spotting. The intentional obfuscation of words and phrases to evade manual or automatic checking often makes detection difficult. Obfuscations such as ni9 9er, whoopiuglyniggerratgolberg and JOOZ make it impossible for simple keyword spotting metrics to be successful, especially as there are many permutations to a source word or phrase. Conversely, the use of keyword spotting could lead to false positives.

Difficult to track all racial and minority insults. One can make a reasonably effective abuse or profanity classifier with a blacklist (a collection of words known to be hateful or insulting), however, these lists are not static and are ever changing. So a blacklist would have to be regularly updated to keep up with language change. In addition, some insults which might be unacceptable to one group may be totally fine to another group, and thus the context of the blacklist word is all important (this forms the motivation for the work by [18]).

Abusive language may actually be very fluent and grammatical. While there are many examples on the internet of abusive language being very noisy, such as in Add another JEW fined a bi$$ion for stealing like a lil maggot. Hang thm all., which can be a helpful signal for an automated method, there are actually many cases where abusive language, or even more specifically hate speech, is quite fluent and grammatical. For example:I am surprised they reported on this crap who cares about another dead nigger?

Abusiveness can be cross sentence boundaries. In the sentence Chuck Hagel will shield Americans from the desert animals bickering. Let them kill each other, good riddance!, the second sentence which actually has the most hateful intensity (them kill each other) is dependent on the successful resolution of them to desert animals which itself requires world knowledge to resolve. The point here is that abusive language is not limited to just the sentence. In some cases, one has to take the other sentences into account to decide whether the text is abusive or carries incidences of hate speech.

Sarcasm. Finally, we noted cases where some users would post sarcastic comments in the same voice as the people that were producing abusive language. This is a very difficult for humans or machines to get correct as it requires knowledge of the community and potentially even the users themselves:

same thing over and over and over and over day in night and day ’cause i am handicapped and stay home. i hate jews they ran over my legs with their bmw. so i will blast them everyday.. I really hurt them i am so powerful .. If ipost about jews here they all suffer. im sow powerfull bwbwbwbwaaahahahahahah im a cripple but i can destroy them with my posts.. I am super poster. Bwwbwahahahaha noone can find me .. I am chicken so i can post behind yahoos wall of anonymous posters. Bwbwbwbabahahahah i will give him ten thumbs down and slander jews.. Bwbwbwbahahahah..i am adoplh hitler reincarnated.