Google networking error caused outage

"Widespread outages involving several Google services--including search, Google Docs, and Gmail--were caused by an upgrade gone awry inside of Google, according to engineers.

Dmitri Alperovitch, vice president of threat research for McAfee, said that Google this morning attempted to make changes to key Internet routing numbers--known as autonomous system numbers--as part of its ongoing transition from an older networking standard to a newer one called IPv6. An unknown "bug" inside Google's network involving some sort of hardware failure or glitch prevented Internet service providers from finding Google's new ASNs on the Internet--effectively sealing it off from many customers, he said.

Not all Internet users were affected, but some that use larger providers--such as AT&T or Verizon--appeared to be disproportionately hurt because large ISPs "peer" with Google, or interconnect their networks with Google's networks in order to improve speed and reduce bandwith costs, Alperovitch said. Not all customers at those providers were affected, and smaller ISPs that didn't interconnect their networks were able to route around the problem. But just like when a bad car accident shuts down a key highway, the ripple effects were felt elsewhere.
The outage began at 8:13 a.m. PDT, according to McAfee's data, and was fixed by 9:14 a.m. PDT. The issue was discussed inside forums dedicated for ISPs and their engineers, such as the North American Network Operators Group. McAfee's customers reported the issue to the security company, which monitors network traffic for some customers.

....

"We've been working hard to make our services ultrafast and 'always on,' so it's especially embarrassing when a glitch like this one happens. We're very sorry that it happened, and you can be sure that we'll be working even harder to make sure that a similar problem won't happen again," Google wrote."