Who turned the Internet Off?

I started this Monday morning, like any other Monday catching up with the office in NY via Skype. Out of the sudden I lost my internet connectivity. Not the first time it happened, so I figured out it might have been the router or the ISP, so I simply rebooted it – but no luck.

Fifteen minute later the internet is back on its own and I get back on Skype to continue my conversation. To my surprise I find out the person on the other line – all the way in New York City – had just experienced the same connectivity problem. Quickly we found out we were not alone – Twitter had several people complaining of issues all around US and the globe, Gizmodo put a quick article blaming it on Timewarner.

We quickly took a look at the performance monitoring data collected by Catchpoint monitoring stations around the world and discovered a jump in failures across all cities – both on backbone Carriers and Last Mile locations on Timewarner, ATT & Verizon FIOS. The problem started at around 9:15 am EDT and lasted until 9:30 am EDT.

Our probes captured several failures for major web sites, ad serving companies, content distribution networks, Public DNS resolvers… as if someone used some kind of kill switch. Several of the captured traceroutes during this period, showed routes that went nowhere – the routers did not know the paths to take to reach the destination.

From Singapore AWS to a Major CDN (Why not served out of Singapore… not the right topic)

Tracing route to i.cdn.turner.com [8.26.197.254] over a maximum of 30 hops: