Understanding the Internet’s insecure routing infrastructure

Last April, 15 percent of the world's Internet traffic was rerouted through …

As an incident that we reported on last week shows, the Internet routing system isn't as secure as we want it to be. But how bad is it really?

Let's start with a very short introduction into Internet routing. Routing is based on autonomous systems (ASes) exchanging prefixes (ranges of IP addresses) using the Border Gateway Protocol (BGP). Autonomous systems are first and foremost Internet Service Providers (ISPs). However, some end-user organizations swim with the big fish, usually in order to connect to two or more ISPs at the same time. The IP addresses ISPs give out to their customers are aggregated into a relatively small number of prefixes that cover large address blocks, and these prefixes are "announced" or "advertised" over BGP to other ASes. Prefixes make their way from AS to AS, so eventually the entire Internet knows where to send packets with a given destination address.

The term "Border Gateway Protocol" makes more sense in the context of 20 years ago, when the word "gateway" was used for what we today call a router. So BGP is the protocol used between border routers—the routers that sit at the edge of neighboring autonomous systems. ASes connect in a hierarchy that looks a lot like this:

The up/down relationships are between ISPs and their customers; the customer pays the ISP. The dotted lines from side to side are "peering" relationships where traffic is exchanged without money changing hands. The economic models are such that traffic flows up the hierarchy, then sideways, and finally down. Paths that go sideways, then up or down, and then sideways again only happen if someone is giving away free service, and is thus rare.

So packets from AS 6 to AS 5 may follow the path 6 - 3 - 1 - 2 - 5, where AS 6 pays AS 3, which in turn pays AS 1, with AS 5 paying AS 2. So all the ISPs get paid, even though AS 1 doesn't pay AS 2 (or the other way around). However, the path 6 - 3 - 4 - 2 - 5 is not a valid way to get from AS 6 to AS 5. In this case, AS 4 would have to pay AS 2 for this traffic, but AS 3 doesn't pay AS 4 anything so, so AS 4 would be giving away service for free. On the other hand, from AS 6 to AS 8 over the path 6 - 3 - 4 - 8 is fine, because AS 8 is AS 4's customer so AS 8 pays AS 4 for the incoming traffic.

BGP itself is blissfully unaware of all the money issues. In its default state, BGP will trust everything that it hears and happily give away service for free. To avoid this, BGP routers must be configured with filters that make sure only correct information is carried by the protocol. Additionally, prefix advertisements, which are BGP's way of inviting incoming traffic, are only sent in accordance with business relationships.

With knowledge of how each AS is interconnected with other ASes and whether a connection involves a customer/ISP relationship or peering, it's possible to know exactly how each destination throughout the Internet should be reached from any given source. In addition, it's necessary to know which range of IP addresses belongs to which AS. Accounting for rerouting after failures adds some complexity, but is not a problem.

Knowledge of the graph of the network and the AS-prefix relationships would make it possible to create filters that validate the information received over BGP and reject incorrect or falsified information, and there are routing databases where ASes can register this information. Unfortunately, many people fail to do so, and the information that is there is often incomplete. The IETF and the Regional Internet Registries that give out IP addresses and AS numbers are now working on a database and certificate infrastructure that would allow network operators to do exactly this, but we're not there yet.

Where are those servers, anyway?

Network operators simply don't know whether CNN's servers are in Atlanta or Beijing. So when BGP updates come in claiming the latter, ISPs—well, their routers—have very little choice other than to install those updates and start sending traffic in the new direction. 999 times out of 1,000, a rerouting event like this is a routine change in the network or an equally routine repair of a failure. But that last one in 1,000 is incorrect—either a mistake or because of an attack of some kind.

Back in the 1990s, an incident like the one that sent traffic to China would have forced network engineers to scramble for hours debugging and fixing the problem. But these days, such incidents are way too common. As a result, a battery of monitoring systems is available Internet-wide. So the notion of something like this staying under the radar is about as likely as a tank driving down Pennsylvania Avenue without drawing any attention.

This leads to an unpleasant state of cognitive dissonance. On the one hand, it's inconceivable that Internet routing is so gullible. On the other hand, it works well most of the time. Fixing the situation would be complex, expensive, and wouldn't pay off anytime soon.

(I went to my first IETF meeting in 2002, when inter-domain [read: inter-AS] routing security was high on the agenda. I remember having lunch at a pizza place in Atlanta with 20 people from Cisco, furiously drawing AS topologies on our napkins the whole time. At that time, there were two proposals on the table to make BGP more secure: S-BGP [Secure BGP] by BBN and soBGP [secure origin BGP] by Cisco. A lot of fighting about these proposals and the difficulties of fixing the problem and even making any changes at all to BGP made almost a decade pass with little to show for it.)

Don't underestimate the complexities involved in securing Internet-wide routing. What if a certificate used in S-BGP or soBGP expires? If that means that a connection is brought down, good luck downloading the updated certificate.

Routing is a critical real-time system. In such systems, the traditional security model of shutting down when security credentials can't be validated doesn't really work. When the system is working, it's important to use security mechanisms to keep attackers from disrupting it. At the same time, it's essential that these same security mechanisms don't get in the way of fixing the system when it has failed or is about to fail. Unfortunately, existing security models don't strike that balance.

The saving grace of Internet routing is that most ISPs carefully filter what their customers send them. So, if I instruct my BGP router to tell my ISP that I'm the owner of the Windows Update IP address, my ISP should know better and ignore this BGP advertisement. Since there is a business relationship between ISPs and customers, both sides have an incentive to keep up to date with any changes in prefixes that are announced.

Once incorrect information has passed the customer-ISP boundary, however, it will quickly spread over peering connections without finding too much in the way of filtering in its way. Because, as noted above, there is currently no Internet-wide database of authoritative routing information that can be used to generate filters. The only way for ISPs to filter their peers would be to continuously exchange updated information on a one-to-one basis. Because of the continuous churn in customers and the addition of new prefixes, as well as the large numbers of peers big ISPs have, this is simply unworkable.

Reexamining the China routing snafu

So what exactly happened in China that caused 15 percent of the Internet's prefixes—not 15 percent the traffic—to get rerouted to that nation in April? And was it an accident or something more malicious? I wasn't at the China Telecommunications Corporation office to observe the incident as it unfolded, so I can't say for sure whether this was a diabolical plan executed to perfection or a network engineer doing something really, really stupid. But I'm betting on the latter, and not just because of Hanlon's Razor.

One common BGP screw-up is leaking the entire routing table. There are currently some 341,000 prefixes making up the Internet and, in order to be able to reach them all, BGP routers need to have all of these in their routing tables. If, for some reason, a BGP router doesn't have any filters, it will simply send a copy of that full table to all the routers in neighboring ASes that it's connected to.

Leaking a full table is a mistake that happens fairly regularly, and it looks like this is what happened in China. Here's what may have gone down.

When a filter is updated, it can become nonfunctional. Usually, this is caught with a "maximum prefixes" filter of last resort—this kills a BGP session if more than a predetermined number of prefixes is received. But, even without this, such a leak usually isn't too devastating because a detour through (for instance) China means that additional ASes are traversed, and BGP prefers shorter paths over longer ones. This is possible because, for each prefix, the ASes on the way to that destination are recorded in an "AS path."

However, just leaking the full table, or at least a sizable fraction of it, was exacerbated by a curious design decision by China Telecom. That decision made it look to the outside world like China Telecom had also stripped off the AS path from all the prefixes that it had leaked. So it looked to peers of China Telecom that destinations such as CNN were located in China Telecom's network, rather than that CNN was merely reachable through China Telecom.

This is why relatively many ASes started sending their traffic to China. Stripping off the AS paths happens when information in BGP is exported into another routing protocol that is used locally, and then re-exported back into BGP. This practice is considered dangerous because of earlier incidents like the one discussed here. There is also not really any good reason to do it—there are a few not-so-good reasons—and I can't think of any way that this would happen by accident.

So just the leaking of a full or partial BGP table in itself isn't too suspicious, although networks the size of China Telecom should know better. But doing so with the AS paths removed may be construed as a reason for a moderate level of suspicion by those so inclined.

If I were more paranoid, however, I would start looking around the Internet for anomalous prefix/AS combinations that show up randomly for short periods of time. Someone who really wants to intercept traffic would be better off hosting a few servers and a BGP router in a data center somewhere in the well-connected world and then try to see which ISPs fail to filter properly. With an ISP located, a targeted attack could be mounted to reroute traffic for much longer than 18 minutes without being found out. Rerouting North-American prefixes within America would also be much less suspicious than rerouting North-American prefixes to China, not to mention providing a much higher degree of deniability.

While we wait for some form of BGP security to become available, we should all think about what would happen if the addresses of the remote systems we communicate with are rerouted and our traffic intercepted. Encryption and cryptographic authentication such as HTTPS or a VPN protects against this. However, there is a problem with encryption: certificate authorities may not be as trustworthy as they should be. How to work around that is a story for another day.

Iljitsch van Beijnum
Iljitsch is a contributing writer at Ars Technica, where he contributes articles about network protocols as well as Apple topics. He is currently finishing his Ph.D work at the telematics department at Universidad Carlos III de Madrid (UC3M) in Spain. Emaililjitsch.vanbeijnum@arstechnica.com//Twitter@iljitsch

37 Reader Comments

As an incident that we reported on last week shows, the Internet routing system isn't as secure as we want it to be.

What? How so? The incident you reported on was meaningless unless it had some impact on performance, otherwise no one should care. Anything that needs to be sent over any public network securely should be encrypted, and routing has zero to do with cracking AES or whatever someone chooses to use. And anything that isn't encrypted can be relatively trivially snooped at many points anyway. And in fact you should assume at the least governments are or will be sucking up everything they can, not a matter of any particular threat or paranoia, it's just as the cost of doing so falls more of a "might find something of interest so why not" thing. Or on a more directly local and practical level, there's certainly been plenty of stories about people accessing the net through random public WiFi access points (or even company run ones).

What we really should be aiming for is a situation where virtually nothing is ever sent unencrypted. So far there hasn't much of a major push for that even where it's important and trivial (Email) let alone in general (no HTTPS for Ars), and costs still have a ways to fall there. But that's where things should be headed. The only thing that routing should consider is optimizing speed, latency, and efficiency.

Quote:

Encryption and cryptographic authentication such as HTTPS or a VPN protects against this. However, there is a problem with encryption: certificate authorities may not be as trustworthy as they should be.

Perfection isn't required though, just constant improvement. It's significantly more difficult to compromise a general certificate authority then merely tap into traffic somewhere along the net, doing so to a single CA or even multiple ones still won't give the kind of access that a stream of completely clear traffic at core/border routers offers, and it's of course completely ineffective against a private CA (the latter is common in the case of VPNs of course).

Quote:

How to work around that is a story for another day.

I'll look forward to reading that article, as I'm certainly in favor of much more widespread pushing of public/private cryptography for the general population. At a minimum for example I'd like to see that integrated into our general ID/financial tech.

As an incident that we reported on last week shows, the Internet routing system isn't as secure as we want it to be.

What? How so? The incident you reported on was meaningless unless it had some impact on performance, otherwise no one should care. Anything that needs to be sent over any public network securely should be encrypted

The very knowledge that there was something sent is knowledge enough, for security-sensitive applications. As well as timing, amount of data, etc... All sort of things that encryption does nothing about.

And I am not even talking about national security here, merely about the potential effect on share price of knowing that two corporations that are officially competitors are exchanging 20Gb encrypted files every day...

The very knowledge that there was something sent is knowledge enough, for security-sensitive applications. As well as timing, amount of data, etc... All sort of things that encryption does nothing about.

But there are plenty of countermeasures known against stuff like timing attacks available higher up the stack, like randomized padding. And if everything thing is encrypted, or a constant stream of dummy data is sent, it's even more difficult to make sense of merely the binary condition of "something was sent or not".

The issue is that there are a lot of places where a really serious malicious attacker can tap into traffic for a target. Trusting the routing layer to keep you secure is a big mistake in my opinion.

Good article showing the BGP overview, but I agree with the first poster. If we didn't get much of a slowdown and nothing broke, how was BGP compromised at all? The protocol gives no guarantee of data security, simply routing reliability.

The incident you reported on was meaningless unless it had some impact on performance, otherwise no one should care.

I fail to see how you can brush off this incident - surely a complete lack of ability to obtain transit to the desired destination is about as serious a performance problem as is possible?

xoa wrote:

Or on a more directly local and practical level, there's certainly been plenty of stories about people accessing the net through random public WiFi access points (or even company run ones).

But that's the user's decision, not a problem with one of the fundamental protocols that keeps traffic flowing for everyone on the Internet.

xoa wrote:

But there are plenty of countermeasures known against stuff like timing attacks available higher up the stack, like randomized padding. And if everything thing is encrypted, or a constant stream of dummy data is sent, it's even more difficult to make sense of merely the binary condition of "something was sent or not".

So, in one post you suggest that the only metric people should care about is performance, and in the next you advocate encryption of all connections, and a constant stream of dummy data??? Those statements are at complete odds with each other.

xoa wrote:

The issue is that there are a lot of places where a really serious malicious attacker can tap into traffic for a target. Trusting the routing layer to keep you secure is a big mistake in my opinion.

Yes, but ensuring that traffic gets to the intended destination must be a priority, without that you have nothing. Simply because there are other areas of weakness does not excuse the need to resolve this issue: it's not simply about data security - though that's certainly a serious concern (your suggestion that every other Internet protocol needs to be overhauled to add encryption, or that TCP needs to be modified in this fashion is, sadly, a pipe-dream at this stage, and a whole other can of worms) - but about ensuring basic continuity of connectivity for users.

As an incident that we reported on last week shows, the Internet routing system isn't as secure as we want it to be.

What? How so? The incident you reported on was meaningless unless it had some impact on performance, otherwise no one should care. Anything that needs to be sent over any public network securely should be encrypted, and routing has zero to do with cracking AES or whatever someone chooses to use. And anything that isn't encrypted can be relatively trivially snooped at many points anyway. And in fact you should assume at the least governments are or will be sucking up everything they can, not a matter of any particular threat or paranoia, it's just as the cost of doing so falls more of a "might find something of interest so why not" thing. Or on a more directly local and practical level, there's certainly been plenty of stories about people accessing the net through random public WiFi access points (or even company run ones). ...

Very nice theory, except that in the real world it fails. Very few people would answer this question affirmatively, probably you included: "Would you be willing, right now, to have every packet sent in or out of your network published in a public stream trivially available to anyone on the internet who wanted it? And by anyone, I mean *anyone*."

In theory, theory and practice are the same. In practice, they are different.

Thank you to the Author for attempting to explain this. It really creates a scale in the mind of just how huge and even more powerful the Internet's infrastructure must be. It's also interesting to know that it's not a superhighway at all. It's one big effing toll road!

The author fails to mention that the IETF continues to work on secure routing via the SIDR (Secure Inter Domain Routing) working group. The RIRs are actively involved in the development of RPKI (Routing PKI) as part of the work SIDR is doing. The US government is also involved via the DHS and other organizations and it actively pushing (via grants) for the development of a routing security mechanism. This does not change the fact that securing BGP is a non-trivial thing but I think it is important for people to understand that a lot of work is being done in this area.

In 20 years the Internet will likely be a very different place with some form of PKI in place across all layers of the network from layer 2 (SEND) to 3 (IPSEC) to 5 and beyond (DNSEC, HTTPS, etc), Whether all of this complexity buys us any real security (as individuals) remains to be seen though....

This is why relatively many ASes started sending their traffic to China. Stripping off the AS paths happens when information in BGP is exported into another routing protocol that is used locally, and then re-exported back into BGP. This practice is considered dangerous because of earlier incidents like the one discussed here. There is also not really any good reason to do it—there are a few not-so-good reasons—and I can't think of any way that this would happen by accident.

This [Source] is probably the most novel approach to traffic thieving that I've heard of.

As an incident that we reported on last week shows, the Internet routing system isn't as secure as we want it to be.

What? How so? The incident you reported on was meaningless unless it had some impact on performance, otherwise no one should care. Anything that needs to be sent over any public network securely should be encrypted, and routing has zero to do with cracking AES or whatever someone chooses to use. ...

Quote:

Encryption and cryptographic authentication such as HTTPS or a VPN protects against this. However, there is a problem with encryption: certificate authorities may not be as trustworthy as they should be.

Perfection isn't required though, just constant improvement. It's significantly more difficult to compromise a general certificate authority then merely tap into traffic somewhere along the net, doing so to a single CA or even multiple ones still won't give the kind of access that a stream of completely clear traffic at core/border routers offers, and it's of course completely ineffective against a private CA (the latter is common in the case of VPNs of course).

The problem is with Man In The Middle attacks. If you have traffic routed through you can spoof whatever site you want. CA's are supposed to solve this, but if you own a CA that ships with browsers (like CNNIC), then you can generate spoofed certificates for any site you want and the browser will believe it. It isn't necessary to compromise another CA if you own your own like China does - and yes this works even against private certificates.

We need to work on improving the situation with CA to make this is more difficult, but as you said, perfection is something we will never have, so security should be multilayered. MITM attacks are always harder to protect against than other types, so having a routing system that lets malicious people setup a MITM situation whenever they request it isn't a good thing.

As an incident that we reported on last week shows, the Internet routing system isn't as secure as we want it to be.

What? How so? The incident you reported on was meaningless unless it had some impact on performance, otherwise no one should care. Anything that needs to be sent over any public network securely should be encrypted, and routing has zero to do with cracking AES or whatever someone chooses to use. And anything that isn't encrypted can be relatively trivially snooped at many points anyway...

Ok, let say China in coop with some large industry that compete's with the US does this on a day that might be important to the US businesses when it comes to traffic...

You dont see the potential to game the system so that the US businesses lose out? What the data contains isn't as imporant as making sure that data never get's to where it needs to...

Maybe this example sucks, but I cant quickly come up with an other. Google and MS are launching new serach-realted services on thier seach sites that directly compete. Google in cohoots with the Chinese telco, cuases all of Bing traffic to never actually reach bing or to also just end up at google... Users's dont get it at first, and just thinks its some local ISP or Windows or IE bug but the damgae is done...

Thank you to the Author for attempting to explain this. It really creates a scale in the mind of just how huge and even more powerful the Internet's infrastructure must be. It's also interesting to know that it's not a superhighway at all. It's one big effing toll road!

Back in the 1990s, an incident like the one that sent traffic to China would have forced network engineers to scramble for hours debugging and fixing the problem. But these days, such incidents are way too common. As a result, a battery of monitoring systems is available Internet-wide. So the notion of something like this staying under the radar is about as likely as a tank driving down Pennsylvania Avenue without drawing any attention.

If there's a battery of monitoring systems that guarantees that this sort of problem will be quickly detected, what's the security problem?

Does the full routing table leak mean that this whole chain of routs is sent to for example a Canadian ISP, i.e. that the traffic will then flow through china just because the "china-router1-as1" -> "china-router2-as1" gets copied to the Canadian ISPs table?

And does the AS-stripping mean that the table sort of looks like;"china-router1-as1" -> "china-router2-as1" -> "us-router-1-as1" -> "uk-router1-as1" from an outside perspective? I.e. even though you traverse AS:es the Canadian ISPs routers would believe it was all in AS1 from the start?

Does the full routing table leak mean that this whole chain of routs is sent to for example a Canadian ISP, i.e. that the traffic will then flow through china just because the "china-router1-as1" -> "china-router2-as1" gets copied to the Canadian ISPs table?

If the Canadian ISP then talks to the US ISP, it would see for the BBC throug the US ISP:

2 3

The Chinese ISP would also see

2 3

So if the Chinese ISP has a regular leak, the Canadian ISP sees:

1 2 3

However, 2 3 is shorter than 1 2 3 so the route through China may be ignored. (If the Canadian ISP is a customer of the US one and a peer of the Chinese one, it may prefer to go through China anyway as peering doesn't cost any money.)

But if China says:

1

then the path is shorter and bad things will happen.

[/quote]I.e. even though you traverse AS:es the Canadian ISPs routers would believe it was all in AS1 from the start?[/quote]Yes, although it's doubtful whether the packets still get to their destination in this case.

I think the answer is IPv6! With IPv6 the new peering agreements and infrastructure has yet to be set up. It's a PITA to try to upgrade a legacy system like IPv4 and billions of client machines. But IPv6 peering could take some lessons of DNSSEC and twenty+ years of internet primetime.

Then all the critical .gov and .mil traffic could be transferred to a more secure IPv6 based infrastructure.

Lenin, I was saying only that since new peering agreements have to be made, it is much easier to introduce additional security extensions on a technical level for IPv6 based traffic. There is no "don't touch what's working" hindrance.

There have been claims by some people that they could bring down Ciscos quickly because of bugs in their IOS (not to be confused with iOS), and there have certainly been unintended problems in this area that make it hard to dismiss such claims out of hand.

The most devious attack that I had in mind when writing the story, but didn't explain, which is more recent than the L0pht group AFAIK, is as follows. The attacker needs two connections to the internet. On the first, they announce a sub-block (more specific prefix) out of someone else's address range. They add to the AS path the ASes from the attacker to the victim over the other path, which means that those ASes won't see the fake announcement. Packets then come in over the first connection and can be sent out to the victim over the second connection so they reach the victim, making it possible to snoop on this traffic. The victim sees nothing out of the ordindary in their BGP tables.

But of course monitoring systems will see this, unless their ASes are also included in the AS path of the fake prefix. Any ISP who allows a customer to do this should be considered criminally negligent in my non-lawyer opinion.