AmazonFail: An inside look at what happened

I’ve spoken to an Amazon.com employee who works closely with the systems involved in the glitch. The employee asked me not to share his name because of company policies on talking with the media.

On Sunday afternoon at least 20 Amazon.com employees were paged alerting them that items, possibly many, were incorrectly being flagged as adult. The employees also received links to the Twitter discussion AmazonFail.

Thousands of people were angry that gay-themed books had disappeared from Amazon’s sales rankings and search algorithms. The number of Tweets on Sunday afternoon that had the term “AmazonFail” surpassed even those with the words “Easter” or “Jesus.”

By this time, Amazon.com had upgraded the problem to Sev-1. (Amazon.com breaks down its operational issues in terms of severity levels. Sev-3 means a problem affects a single user. Sev-2 is a problem that affects a company, or a lot of people. Sev-1 is reserved for the most critical operational issues and often are sent up the management chain to the senior vice president level.)

“People got pulled away from their Easter thing when this whole thing broke,” the employee said. “It was just a screwup.”

Amazon.com employees are on call 24/7, and many began working on the problem from home. It didn’t take much digging to realize that there was a data error.

Amazon managers found that an employee who happened to work in France had filled out a field incorrectly and more than 50,000 items got flipped over to be flagged as “adult,” the source said. (Technically, the flag for adult content was flipped from ‘false’ to ‘true.’)

“It’s no big policy change, just some field that’s been around forever filled out incorrectly,” the source said.

Amazon employees worked on the problem well past midnight, and then handed it over to an international team, he said.

Seattle-based Amazon.com sells millions of items, so the 57,000 affected represent just a tiny portion of the company’s selection. But Amazon’s perception problem was enormous, and aggravated by the company’s official description of the problem as a “glitch.”

The source wanted it known that mistakes do happen at Amazon.com — they’re all human.

“Most everyone at one point who works with catalog systems has broken some piece of the catalog,” the source said.

In fact, employees sometimes tell war stories over lunch where the conversation can go something like this: “Oh yeah, I remember when I broke a portion of France and all this stuff happened.”

Usually, mistakes get corrected pretty quickly, the source said.

And there you have it from a source who watched the situation unfold within his company and on Twitter.