Google is investigating an unorthodox routing of internet traffic that on Monday sent traffic bound for its cloud services instead to internet service providers in Nigeria, Russia and China.

The routing problems persisted for about two hours before they were fixed, says Alex Henthorn-Iwane, vice president of product marketing for the security company ThousandEyes.

The fact that it affected such a large swath of Google's networks makes it unlikely the routing was simply an error, especially because it involved network providers within Russia and China, Henthorn-Iwane says.

"It's not a mistake," he says. "There's nothing about this that suggests that this was a mistake."

Traffic bound for Google services went through Nigeria's MainOne, China Telecom and Russia's TransTelekom. In an update on its Cloud Platform service site, Google says its services operated as expected, although that contradicted what networking monitoring companies observed. The cause of the issue appears to be external, Google says.

Google says it is investigating the re-routing of traffic bound for its services.

"We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence," the company says.

BGP is an open standard used by all ISPs. But routing traffic can be a delicate process, and mistakes can end up in outages. Also, there are concerns that nation-states have been tampering with BGP in order to watch internet traffic for surveillance purposes (see: Who's Hijacking Internet Routes?).

Criminals have also taken an interest in BGP. Earlier this year, an attacker successfully created a BGP leak, seized control of IP addresses and intercepted cryptocurrency wallet data (see: Cryptocurrency Heist: BGP Leak Masks Ether Theft).

Some Traffic Ended in China

The Monday traffic routing problem for Google meant that some U.S. users attempting to reach Google may have been bounced through other countries before reaching Google's servers. In some cases, it appears that traffic that went through China Telecom didn't actually end up connecting with Google at all.

China blocks many Google services, so it is possible that the country's so-called Great Firewall, which blocks access to many services as part of the country's censorship regime, just dropped the traffic. It's also possible that too much traffic may have hit China Telecom at once, causing a jam.

ISPs "announce" new internet traffic routes between networks, known as autonomous systems - ASes - using BGP. Those announcements are then often re-announced or propagated by other ISPs, which is intended to ensure that worldwide routing stays efficient while networks come online or others go down.

But computer security experts have long warned that BGP is particularly vulnerable to malicious manipulation because it's an open system. It is possible for ISPs to reject BGP announcements from other ISPs, and many now more closely watch announcements that may be errant, for security reasons.

"It's the irony that the very openness, agility and resilience that the internet displays also creates a vulnerability," Henthorn-Iwane says.

Ongoing Security Risk

Only Google should be announcing its traffic routes. But what happened on Monday is that someone or some entity pretended it had authority to announce new routes to Google services.

BGP is largely an honor system, Henthorn-Iwane says. Network operators hope that other network operators won't make unauthorized BGP announcements on their behalf. But there are no real mechanisms in place to stop that from happening, leaving operators dependent on detecting mistakes on their own.

Alan Woodward, a professor of computer science at the University of Surrey, describes fundamental internet protocols such as BGP and DNS as being "the soft underbelly of the web," in desperate need of a security overhaul. To date, however, the political and business will to fix these legacy infrastructure problems hasn't been found.

Woodward tells Information Security Media Group that the BGP problems stem from the internet being a network of networks, and no one having designed any of its core protocols with security in mind. "Security was never thought to be an issue as no one envisaged it being used as it is now," he says.

"BGP is totally based upon trust at present and if that is broken - by mistake or deliberately - then routing can be subverted. There are initiatives to try to secure BGP, such as Secure Inter-Domain Routing, but they will take a long time to be universal."

As specified by the Internet Engineering Task Force, Secure Inter-Domain Routing is an initiative - not a standard - that would enable the creation of infrastructure that would allow an entity "to verifiably assert that it is the legitimate holder of a set of IP addresses or a set of autonomous system numbers," according to the IETF's overview.

Pending such fixes, BGP security remains an ad hoc affair. "Many countries have the luxury of having someone watch for unusual routing announcements and to correct as needed," Woodward says. "Smaller countries - some may have a handful of ISPs at best - do not."

Rob Joyce, the National Security Agency's senior adviser for cybersecurity strategy to the director, says via Twitter: "I hope this latest fiasco of traffic rerouting through China is the wake-up call for all of us to get serious about addressing the massive and unacceptable vulnerability inherent in today's BGP routing architecture."

Rogue Announcements

Henthorn-Iwane says that the errant BGP announcement affected 150 or so "prefixes," which are essentially blocks of IP addresses. Each Google prefix contains as many as 8,000 individual IP addresses, he says.

It's not clear where the rogue announcement originated. But Henthorn-Iwane later wrote on Twitter that it could have occurred at an internet exchange in Lagos, where China Telecom and MainOne peer. ISPs often connect together at centralized points, called internet exchanges, to save on costs. Such arrangements are known as peering.

Once the routing announcement was made, it appears that China Telecom then broadcast it to the Russian ISP TransTelekom, according to BGPmon, which is part of Cisco's OpenDNS.

The tainted route was then picked up by other ISPs, including NTT and Cogent, before it was eventually fixed, Henthorn-Iwane says.

Defense: Encryption

Traffic hijacking poses security risks, but there is a known defense against anyone being able to study intercepted data: encryption.

Connections between a user and a Google service are encrypted using Transport Layer Security, or TLS, which is often referred to as SSL, short for Secure Sockets Layer. An ISP that's helping to transit traffic would see the originating IP address and know the destination, but the content itself would be encrypted.

But intelligence services around the world actively work to break encryption, and there are certain kinds of attacks designed to break SSL. Decrypting traffic, however, is thought to be impossible for anyone who doesn't possess the necessary TLS keys, which, in theory, should be well protected.

Today's computers aren't powerful enough to rapidly calculate decryption keys, but quantum computing is expected to change that. Experts suspect that well-resourced intelligence agencies may be recording traffic now in the hope that once quantum computing become more prevalent and cost-effective, this historical traffic can be decrypted and analyzed.

MainOne Takes the Blame

Meanwhile, later on Tuesday, Nigerian ISP MainOne took the blame for the BGP routing problem, and said it had been fixed less than 90 minutes after it happened.

We have investigated the advertisement of @Google prefixes through one of our upstream partners. This was an error during a planned network upgrade due to a misconfiguration on our BGP filters. The error was corrected within 74mins & processes put in place to avoid reoccurrence

"This was an error during a planned network upgrade due to a misconfiguration on our BGP filters," the ISP said.

Despite the mea culpa, many continue to urge caution. "I'll take the Nigerians at their word that they didn't intentionally hijack the BGP routes," says Jake Williams, founder of Rendition Infosec, a security consultancy based in Augusta, Georgia. He's also an instructor at the SANS Institute and a former operator with the NSA's Tailored Access Operations unit.

"But if I were China and wanted plausible deniability, I'd use an ISP in a country known for corruption," he says. "TL;DR don't take everything at face value."

About the Author

Kirk is a veteran journalist who has reported from more than a dozen countries. Based in Sydney, he is Managing Editor for Security and Technology for Information Security Media Group. Prior to ISMG, he worked from London and Sydney covering computer security and privacy for International Data Group. Further back, he covered military affairs from Seoul, South Korea, and general assignment news for his hometown paper in Illinois.

Operation Success!

Risk Management Framework: Learn from NIST

From heightened risks to increased regulations, senior leaders at all levels are pressured to
improve their organizations' risk management capabilities. But no one is showing them how -
until now.

Learn the fundamentals of developing a risk management program from the man who wrote the book
on the topic: Ron Ross, computer scientist for the National Institute of Standards and
Technology. In an exclusive presentation, Ross, lead author of NIST Special Publication 800-37
- the bible of risk assessment and management - will share his unique insights on how to:

Understand the current cyber threats to all public and private sector organizations;

Develop a multi-tiered risk management approach built upon governance, processes and
information systems;

Enter your email address to reset your password

Already have anISMG account?

Forgot Your Password Message:

Contact Us

Already have anISMG account?

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing inforisktoday.asia, you agree to our use of cookies.