Imposing order on the Xbox 360’s worst trolls

Stephen Toulouse reflects on his years as Xbox Live’s head “beat cop.”

Trying to moderate a large community on the Internet is not an easy job. Anyone who tries to enforce basic rules of civility and respect in an open Internet community of any decent size is fighting a never-ending battle against John Gabriel’s Greater Internet Fuckwad Theory, which turns usually normal people into complete, well, you know.

Those problems are only compounded on a service like Xbox Live. Anyone who has joined a public match in a first-person shooter on the Xbox 360 knows the platform is infamous as a haven for antisocial, antagonistic, hate-spewing preteen boys pumped up with adrenaline from shooting virtual people in the virtual face.

But Stephen Toulouse, who served as Xbox Live’s head of enforcement for five years before departing earlier this year, said managing Xbox Live’s tide of miscreant behavior was just a matter of bringing consistent punishment to those who break the rules.

“The reason bad behavior on the Internet occurs is because of a lack of consequences,” he told Ars Technica in an interview about his tenure at Microsoft. “I grew up in the world of arcades, and playing sports in a local park, where if I spouted off about some kid to his mother, there could be immediate and physical retribution. There were consequences [for] that behavior.”

The beat cop

Toulouse’s job at Microsoft was to bring consequences to the virtual world, using a team of hundreds of “enforcers” around the world to make sure Xbox Live’s tens of millions of users were sticking to the Xbox Live Code of Conduct. That meant stopping game-breaking problems like unauthorized cheating and console modifications, but also handling interpersonal issues like harassment, abuse, threats, obscenity, defamation, spam, and “racial, ethnic, or religious slurs.”

While many users are under the impression that the entire enforcement process is automated, Toulouse said every complaint sent in by an Xbox Live user is actually reviewed by a human, usually in much less than the targeted 24-hour window. Toulouse said he couldn’t discuss the forensic “secret sauce” his team used to confirm that those complaints were genuine, but he assured me the enforcement tools lead to an error rate that was “ridiculously low.”

They also allowed Toulouse and his team to confront people with the evidence of their misdeeds before bringing down punishments like temporary suspensions or permanent system bans. “It really is one of those scenarios where, when you have to answer questions, it was like ‘Yeah, we really did catch you,’” he said. “They just want to know they were caught.”

Enforcement wasn’t just a matter of reacting to complaints, though. Sometimes, Toulouse’s enforcers would serve as “beat cops” on the Xbox Live scene by jumping in to do spot checks in games that tended to contain the most problem players. More than actually catching rulebreakers red-handed, Toulouse said this kind of active Internet policing provided a valuable visible reminder for players that they needed to stay on their best behavior.

“I can’t tell you how many times I joined games and people would quiet up or go away. Those are the people who were misbehaving... I never had a problem with the entire game going silent, but I’d join a game and then hear one of the players go, ‘Oh, now you shut up’ to the other player.”

Problem parents

But dealing with problem players was nothing compared to dealing with certain parents, who “might have a certain set of expectations and perceptions and they may not match reality,” Toulouse said. Even parents that were very conscientious about limiting a child’s access to the Internet on the family computer often didn’t realize that today’s game consoles were similar general purpose devices. “So they put a game console in their kids room, without quite realizing that if they don’t somehow lock down the router or lock down the Internet connection at night, I guarantee you the child is doing something at 2am that the parent would be shocked by,” Toulouse said.

“One of the biggest problems I ran into is what we called the ‘Christmas morning problem,'” he continued. “We had these great parental controls on the Xbox, and other consoles have them too, but the parent hands the box to the child and says, ‘OK, fire it up,’ and the kid creates an adult account. Six months later, that kid does something horrible and gets punished. The parent comes to us and says, ‘What did my child do?’ and they are shocked and appalled. We tell them about the parental controls and they’re very excited, but they never knew… I think the industry has a long way to go still in educating parents about what has become a general purpose device and how much it interacts with the Internet.”

Changing norms

When Toulouse started in 2007, there was a rule in place banning all discussion of sexual orientation, gay or straight, from the service entirely. The rule’s main purpose was to provide some protection against the extremely common, extremely pejorative use of terms like “gay” as a slur against an opponent in an Xbox Live match, Toulouse said, partly on the theory that such terms just “didn’t belong in a gaming context.”

This kind of automatic punishment started to become untenable, though, when players began complaining about facing punishment for referring to their own sexual orientation as a matter of pride in Gamertags like “theGAYERgamer” and in other areas of the service. So, in early 2010, Toulouse and his team worked with GLAAD to modify the policy to allow for certain uses of sexual orientation terms, while still limiting harassment.

While the blanket rule against sexual orientation discussion might have made sense when Xbox Live was primarily an online gaming service, Toulouse said, things weren’t quite as clear-cut once the service started incorporating more general social networking features like Twitter and Facebook. “Suddenly it became sort of archaic and silly to [punish people who would] just state ‘I love my boyfriend,’ and [have] that person be male,” he said. “The vast majority of [the use of gay terms] was still pejorative, but it was one of those things where you take a step back and say, ‘You know, the population has changed, the function of the service has changed slightly, so we need to go ahead and take on the effort to divine whether something is pejorative or not.’”

And as the Xbox Live player base grew over the years to include more casual players, Toulouse said he found his team had to scale up its capabilities faster than normal to handle an increased rate of complaints. “[The hardcore players] might not complain, and it leads you into a false sense of where people are having safe experiences,” he said. “You find in the hardcore world people are more tolerant of miscreant behavior,” he said. “They either are trash talkers themselves or don’t view that as necessarily against the rules, even though it is. But you start to throw in the casual gamer that plays the occasional Modern Warfare 3 match, and that population expects that there be repercussions for breaking the rules.”

As the public face of Xbox Live enforcement, Toulouse said he occasionally ran into disgruntled players begging to get their accounts unbanned. For the most part, though, he said the feedback he and his team got at public events was overwhelmingly positive, despite the costs it imposed on them.

“If you think about it, it’s sort of a tax,” he said of the enforcement role. “Resources that went into my team are resources that don’t go into doing new things in the avatar marketplace, for instance… People at the end of the day want to have a safe and enjoyable experience, and I don’t see how you can increase those players’ experience without some form of enforcement or contravention.”

Promoted Comments

I appreciate what these guys do, it's a tough mountain to climb. For me, online gaming is isolated to friends only. Public matches on any service are just not worth the grief. It seems to me the only enjoyment to be had is "killing all the n00bs." It's practically impossible to find similarly skilled players (the games fault) in most games. I'm glad so many games are including online co-op, that's the future for online gaming for people like me.

I appreciate what these guys do, it's a tough mountain to climb. For me, online gaming is isolated to friends only. Public matches on any service are just not worth the grief. It seems to me the only enjoyment to be had is "killing all the n00bs." It's practically impossible to find similarly skilled players (the games fault) in most games. I'm glad so many games are including online co-op, that's the future for online gaming for people like me.

Why not record all in-game audio (as multiple MP3s - one for each gamer) and if someone complains then go back and review it. After 3 days, delete the old MP3s. Better yet, feed it through an automated text-to-speech processor and if it has banned words on it then flag it for review, this way they don't have to wait for complaints to come in.

Why not record all in-game audio (as multiple MP3s - one for each gamer) and if someone complains then go back and review it. After 3 days, delete the old MP3s. Better yet, feed it through an automated text-to-speech processor and if it has banned words on it then flag it for review, this way they don't have to wait for complaints to come in.

I assumed that was the part of their enforcement mechanism that they "couldn't talk about".

Why not record all in-game audio (as multiple MP3s - one for each gamer) and if someone complains then go back and review it. After 3 days, delete the old MP3s. Better yet, feed it through an automated text-to-speech processor and if it has banned words on it then flag it for review, this way they don't have to wait for complaints to come in.

Even 3 days worth of record will require immense space. Even then it won't stop people from complaining that 3 days are too short.

Automated speech-to-text tends to be inaccurate and opens MS of false red flagging people; with all legal issues that goes along for banning people incorrectly.

I got tired of sending complaints about players using slurs so I now only play while in a party with the mic cable plugged into the controller*, even if I'm not on the mic. That pretty much cuts out everything except for when people send me death threats after BF3 matches.

*I still can't believe that there is no way, in some games at least, to turn off all voice communications. I use a headset and don't want the wails of little dumbasses echoing around me in stereo.

I have to confess, though, that I still don´t understand why anyone would want to publicly emphasize his sexual preferences in a virtual gaming environment.

I mean, people are there for fun and games (literally), why bring up subjects that don´t belong there, such as nationality, ethnic background, sexuality etc.?It´ll only irritate other players and won´t do anything to enhance the actual gaming experience.

Imagine a group of people sitting together playing a cardboard game, and one of them keeps shouting, "Yeah, black power!!!!"! or "Woot, I like blowjobs, watch out...."..... it would make for a, shall we say, rather bizarre scenario.

And that doesn´t even pertain to an obviously pejorative use of certain (slang) words -- something that always spoils the fun, too.

Why not record all in-game audio (as multiple MP3s - one for each gamer) and if someone complains then go back and review it. After 3 days, delete the old MP3s. Better yet, feed it through an automated text-to-speech processor and if it has banned words on it then flag it for review, this way they don't have to wait for complaints to come in.

Would this exclude party chat audio? I'd hate to get punished for calling an old college friend the ass pirate he is.

I've never reported anyone for abusive language though. I just take it as part of the game and mute them if they are that bad but most of the time I just laugh at them.

Better yet, feed it through an automated text-to-speech processor and if it has banned words on it then flag it for review, this way they don't have to wait for complaints to come in.

Automated speech-to-text tends to be inaccurate and opens MS of false red flagging people; with all legal issues that goes along for banning people incorrectly.

Bolded the relevant phrase.

Using text-to-speech by itself would indeed be a legitimately terrible idea (last I checked cbs.com wouldn't allow you to refer to Senator Coons by name, and AOL sites censor out the word "sucking" even if you're talking about a lollipop), but using it to flag something for review by a human being isn't an inherently bad solution.

“The reason bad behavior on the Internet occurs is because of a lack of consequences,” he told Ars Technica in an interview about his tenure at Microsoft. “I grew up in the world of arcades, and playing sports in a local park, where if I spouted off about some kid to his mother, there could be immediate and physical retribution. There were consequences [for] that behavior.”

Abso-fucking-lutely dead on the money.

This is the biggest problem with online communities, IMO. People really do just say: "Fuck it, it doesn't matter what I say, nobody can do anything about it."

Whether it's people who lose their temper or people who are just bitter jerks, raising civil discourse as the standard and norm in places that lack consequences for poor actions or words is impossible.

I've never really had much trouble in XBL to be honest. You have to know how to break the cycle of stupidity and show by example that acting like a douche is just plain embarrassing for them. Just like troll on the internet they try and get other people upset to justify to themselves their behavior, addicts do that as well, "come get drunk with me" but if they're the only one getting "drunk" they feel embarrassed. You also have to treat people with respect, even idiots, often they're so desperate for that they'll quit as soon as you show them a bit of dignity.

I think people find so much bad people on XBL becuase in a way it's a reflection of themselves and how they view others, especially strangers. A lot of what the reality around you is is what you create.

I could care less about taunting and smack talk, cuss words and berating. What I do hate though are the preteens that get on the game and squeek in my ear. Sticks and stones may break my bones, but words will never hurt me... unless they are at a pitch that causes my ear drums to bleed. Instant mute regardless if they are being perfectly civil or not.

As I understand it, game chat still continues to be foul on public games. Have they cleaned it up at all, with regard to the racism and hate speech?

Meh. As of a year or two ago, not much. Thing is, it would be worse if they weren't trying.

SirOmega wrote:

Why not record all in-game audio (as multiple MP3s - one for each gamer) and if someone complains then go back and review it. After 3 days, delete the old MP3s. Better yet, feed it through an automated text-to-speech processor and if it has banned words on it then flag it for review, this way they don't have to wait for complaints to come in.

I'm guessing you're understimating the amount of storage space this would require. Or, going the text-to-speech route, the processing power (and storage space, for the review of all conversation within a game/timeframe that's flagged).

But you figure you've got a couple/few million people, even at high-compression rates, you're talking about terabytes upon terabytes of audio. Probably petabytes. That's what's next, right?

Those problems are only compounded on a service like Xbox Live. Anyone who has joined a public match in a first-person shooter on the Xbox 360 knows the platform is infamous as a haven for antisocial, antagonistic, hate-spewing preteen boys pumped up with adrenaline from shooting virtual people in the virtual face.

I just wish to thank the author for this particular paragraph. Bwahahaha

I got tired of sending complaints about players using slurs so I now only play while in a party with the mic cable plugged into the controller*, even if I'm not on the mic. That pretty much cuts out everything except for when people send me death threats after BF3 matches.

*I still can't believe that there is no way, in some games at least, to turn off all voice communications. I use a headset and don't want the wails of little dumbasses echoing around me in stereo.

You can (on the Xbox 360 at least).. Check out the settings under Settings > Preferences > Voice. You can disable different voice modes or turn the voice volume to 0. You can also select individual players in game and mute them if they annoy you. Another, more temporary, trick would be to create a Party by yourself and leave the party chat on. You won't be able to hear the other players.

You can (on the Xbox 360 at least).. Check out the settings under Settings > Preferences > Voice. You can disable different voice modes or turn the voice volume to 0. You can also select individual players in game and mute them if they annoy you. Another, more temporary, trick would be to create a Party by yourself and leave the party chat on. You won't be able to hear the other players.

++. While I like gaming online (XB Live), and generally have fun in public matches for the games I like to play I have zero interest in talking to anyone or hearing anyone, for any reason. I disabled voice chat in preferences ages ago and haven't missed it.

Don't know about XBL lately or shooters, but I've recently been fairly shocked by how civilised GT5 online is on PSN. Randomly chosen lounges generally contain drivers who are taking the game seriously, racing fairly cleanly and are civilised about it. Not sure how much this is the crowd the game attracts, especially now it's no longer the latest hot release, and how much is down to Sony's efforts to police it, but the end result is much better than I'd anticipated, and better than my few forays online with the original Forza, many years ago.

I got tired of sending complaints about players using slurs so I now only play while in a party with the mic cable plugged into the controller*, even if I'm not on the mic. That pretty much cuts out everything except for when people send me death threats after BF3 matches.

*I still can't believe that there is no way, in some games at least, to turn off all voice communications. I use a headset and don't want the wails of little dumbasses echoing around me in stereo.

Best method I've found if I don't want to hear anyone is to create a party and set it to private/invite only.

Invite you BF3 squad/team mates in and you are set. Anyone in the party that you've invited that is being nasty you can kick from the party.

Usually when I'm by myself I just kill my mic. The squad I'm in rarely talks to randoms.

I attended the panel for these guys at PAX a couple years ago. It was pretty interesting and I'm really curious how their "secret sauce" works, but I guess unless I send them a resume, I'll never know. One thing they did say, is that if you report someone and it turns up to be valid, your account gets (I'm trying to remember) a rating for valid reporting, so their system will use your reports more often or maybe bring them to the front of the line.

So, report those assholes. You might not see them get in trouble, but you can make it more likely they'll get caught.

Why not record all in-game audio (as multiple MP3s - one for each gamer) and if someone complains then go back and review it. After 3 days, delete the old MP3s. Better yet, feed it through an automated text-to-speech processor and if it has banned words on it then flag it for review, this way they don't have to wait for complaints to come in.

I'm guessing you're understimating the amount of storage space this would require. Or, going the text-to-speech route, the processing power (and storage space, for the review of all conversation within a game/timeframe that's flagged).

But you figure you've got a couple/few million people, even at high-compression rates, you're talking about terabytes upon terabytes of audio. Probably petabytes. That's what's next, right?

Yeah, using MP3 compression for that task would be stupid. You'd only need a fraction of the storage if you used a codec targeted specifically for speech purposes such as Speex. Record at, say, 4Kbits/sec with an 8KHz sample rate. If that's still too much storage, Speex supports bit rates down to 2K.

I have to confess, though, that I still don´t understand why anyone would want to publicly emphasize his sexual preferences in a virtual gaming environment.

I remember as a young lad growing up on the rough and tumble playground at elementary school that I was the victim of a joke or two. When I talked to my dad about it, he told me something very important that applies to the section I quoted: "If something bothers you, never, EVER tell anybody. If you do tell them, you will be hounded mercilessly." Is that true? In my experience, absolutely. Should it be so? Probably not, but welcome to the herd mentality and the wonders of anonymity.

If it doesn't bother you to broadcast that you prefer men to women, or vice versa, by all means, go to town. Just know that whatever you put out there is ammunition for some random jerk to use against you should he/she sense a weakness.

“If you think about it, it’s sort of a tax,” he said of the enforcement role. “Resources that went into my team are resources that don’t go into doing new things in the avatar marketplace, for instance… "

Maybe I'm looking at this wrong, but this makes me want to be uncivil. Do people really clamor for more stuff to buy in the avatar marketplace?

Most of the games I've played online, the majority were civil. Once they start whipping on the n-word though, I report them and mute them. That's really the only thing that bothers me. And I'm white... but just can't stand to hear it. Oh, and the guys that blare music into the mic.

Kyle Orland / Kyle is the Senior Gaming Editor at Ars Technica, specializing in video game hardware and software. He has journalism and computer science degrees from University of Maryland. He is based in Pittsburgh, PA.