Breaking the Internet

One Amazon Employee’s “Human Error” May Have Cost the Economy Millions

On Tuesday, significant portions of the Internet were crippled when part of Amazon’s cloud, Simple Storage Service, experienced an hours-long outage. Amazon Web Services, the $404 billion company’s cloud-based computing system, has essentially become a backbone of the Internet; even an intermittent outage knocks out service to social-media platforms, web publishers, and other websites. Within several hours, Amazon’s S3 service was back online, but it had lasting ramifications: according to one estimate, it cost S&P 500 companies $150 million, and U.S. financial-service companies $160 million in lost revenue.

Two days after the fact, Amazon offered an explanation for the events that had caused the outage: simple human error.

The company’s S3 team had been working on fixing an issue that caused its billing system to slow down. An employee intended to take a small number of servers offline to briefly fix the problem, but accidentally entered a command incorrectly and removed a large number of servers instead. At 9:37 a.m. P.S.T., the typo caused an outage. “An authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process,” Amazon said in a statement. “Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”

That apparently set off a domino effect: the servers that were unintentionally removed affected other parts of the S3 system, and Amazon had to restart all of the affected systems, and then complete routine safety checks. Other Amazon Web Services that relied on S3 for storage stopped working too. Apps and websites like Venmo, GitHub, and Slack were all affected.

Four long hours after initially breaking the Internet, Amazon’s systems were entirely rebooted. To prevent a simple human error from causing another blackout, Amazon is taking steps like adding “safeguards” to keep engineers from being able to remove such a large server capacity. “While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses,” the company said. “We will do everything we can to learn from this event and use it to improve our availability even further.”

Sundar Pichai

Sundar Pichai, Google’s C.E.O., was born in Chennai, India, immigrating to the U.S. to attend Stanford in 1993.

Photo: By Simon Dawson/Bloomberg/Getty Images.

Sergey Brin

Alphabet president and Google co-founder Sergey Brin was born in Moscow and lived in the Soviet Union until he was six, immigrating with his family to the United States in 1979.

Photo: By FABRICE COFFRINI/AFP/Getty Images.

Elon Musk

Elon Musk, the founder of SpaceX and Tesla, was born and raised in South Africa. He obtained Canadian citizenship in 1989 and briefly attended college at Queen's University in Ontario. He transferred to University of Pennsylvania, in part because such a move would allow him to get an H-1B visa and stay in the U.S. after college.

Photo: By Justin Chin/Bloomberg/Getty Images.

Safra Catz

Safra Catz, who served as co-C.E.O. of Oracle, was born in Israel. She resigned from her executive role in December after joining Donald Trump’s presidential transition team.

Photo: By David Paul Morris/Bloomberg/Getty Images.

Peter Thiel

Trump supporter Peter Thiel, who has expressed support for the president’s executive action restricting immigration from several predominantly Muslim countries, is an immigrant himself. Before he co-founded PayPal and made one of the earliest large investments in Facebook, Thiel moved with his family from Germany, where he was born. In 2011, he also became a citizen of New Zealand, adding a third passport to his growing collection.

Photo: By Roger Askew/Rex/Shutterstock.

Satya Nadella

Born in Hyderabad, India, Microsoft C.E.O. Satya Nadella came to the U.S. to study computer science, joining Microsoft in 1992.

Photo: By Stephen Brashear/Getty Images.

Garrett Camp

Garrett Camp helped co-found Uber. He was born in Alberta, Canada, and now resides in the Bay Area.

Photo: By Justin Lane/EPA/Rex/Shutterstock.

Sundar Pichai

Sundar Pichai, Google’s C.E.O., was born in Chennai, India, immigrating to the U.S. to attend Stanford in 1993.

By Simon Dawson/Bloomberg/Getty Images.

Sergey Brin

Alphabet president and Google co-founder Sergey Brin was born in Moscow and lived in the Soviet Union until he was six, immigrating with his family to the United States in 1979.

By FABRICE COFFRINI/AFP/Getty Images.

Elon Musk

Elon Musk, the founder of SpaceX and Tesla, was born and raised in South Africa. He obtained Canadian citizenship in 1989 and briefly attended college at Queen's University in Ontario. He transferred to University of Pennsylvania, in part because such a move would allow him to get an H-1B visa and stay in the U.S. after college.

By Justin Chin/Bloomberg/Getty Images.

Safra Catz

Safra Catz, who served as co-C.E.O. of Oracle, was born in Israel. She resigned from her executive role in December after joining Donald Trump’s presidential transition team.

By David Paul Morris/Bloomberg/Getty Images.

Pierre Omidyar

The founder of eBay, Pierre Omidyar, was born in France to Iranian parents. He immigrated to the U.S. in the 1970s.

By Ramin Talaie/Bloomberg/Getty Images.

Jerry Yang

Yahoo co-founder Jerry Yang moved from Taiwan to San Jose, California, in 1978, at the age of 10.

by Scott Olson/Getty Images.

John and Patrick Collison

Brothers John Collison and Patrick Collison, twenty-something college dropouts who emigrated from Ireland, co-founded Stripe, a $9.2 billion payments start-up.

By Jerome Favre/Bloomberg/Getty Images.

Adam Neumann

Adam Neumann, raised on an Israeli kibbutz, moved to the U.S. in 2001, after briefly serving in the Israeli army as a navy doctor. Now he’s the chief executive of the $16.9 billion New York-based WeWork, which sublets space to individuals and companies.

by Noam Galai/Getty Images.

Mario Schlosser

The co-founder and C.E.O. of health insurance start-up Oscar, Mario Schlosser, came to the United States from Germany as an international student, receiving his M.B.A. from Harvard.

By Kholood Eid/Bloomberg/Getty Images.

Peter Thiel

Trump supporter Peter Thiel, who has expressed support for the president’s executive action restricting immigration from several predominantly Muslim countries, is an immigrant himself. Before he co-founded PayPal and made one of the earliest large investments in Facebook, Thiel moved with his family from Germany, where he was born. In 2011, he also became a citizen of New Zealand, adding a third passport to his growing collection.

By Roger Askew/Rex/Shutterstock.

Satya Nadella

Born in Hyderabad, India, Microsoft C.E.O. Satya Nadella came to the U.S. to study computer science, joining Microsoft in 1992.

By Stephen Brashear/Getty Images.

Garrett Camp

Garrett Camp helped co-found Uber. He was born in Alberta, Canada, and now resides in the Bay Area.