Brough's writings on the technology, economic and social issues of communications at the intersection of the Internet, telecom and mobility.

March 10, 2014

Internet Peering, an invited talk at this week's OFC 2014

I've been invited to give a talk on Internet Peering at the upcoming Optical conference, OFC 2014 in San Francisco. If per chance you will be attending, it's Wedneday morning at 9:15am. If not, here's a draft of a paper I wrote to go along with my presentation.

Impact of Internet Peering on Network Architectures and Economics

Brough Turner

netBlazr Inc., 18 Bridge St., Watertown, MA USA

brough@netblazr.com

Abstract: The Internet backbone consists of ~6000 independent networks. The technology and economics of how these networks exchange data drives the location of data centers and the location and utilization of high capacity fiber links. We explain how Internet peering works, how it has evolved and trends that will influence future network deployments.

1. Introduction

The Internet is made up of millions of independently controlled networks that, together, support nearly 3 billion users [1]. Most of these networks, e.g. home networks and medium and smaller enterprise networks, pay an Internet Service Provider (ISP) to provide access to any valid Internet address. Some larger enterprise networks pay for separate connections to two or more ISPs. Tens of thousands of ISPs support access and aggregation. Finally, just over 6000 ISP networks [2] form the Internet backbone by exchanging traffic over a sparse mesh of interconnections that use common link protocols and link technology, but two different business arrangements: “Internet transit” and “peering.” With transit, the upstream ISP offers to handle traffic for any Internet address. In peering, operators forward only those packet destined for subscribers on the peer’s network. This difference is key for the economics discussed in section 3.

2. Interconnection Protocols & Technology

All links transport Internet Protocol (IP) data packets with public IP addresses. Peering links (and many transit links) use Border Gateway Protocol (BGP) (currently BGPv4 per RFC 4271) to advertise the IP address ranges for which they will accept traffic. Because BGP is deployed on a link-by-link basis, there is room to negotiate BGP options (of which there are many) in support of the exchange policies of the specific operators.

Typically, each ISP arranges their own data transport circuits to a common meeting point where each ISP has their own router. The actual peering link is usually an Ethernet connection between these routers, either directly or via an in-building peering fabric (effectively an Ethernet switch provided by an Internet exchange operator).

3. Peering economics

For local or regional ISPs, peering economics are dominated by where you are and how much traffic you have. For example, a regional ISP in mid-state Illinois will have limited options for any kind of Internet transit and no local options for peering. Fiber connections from the incumbent telephone company are widely available but expensive. Competing local fiber is rare while long haul fiber routes have relatively few physical access points. Outside of major cities, the rates for Internet transit service can be 10x to 100x greater than at a neutral data center in a major hub like Chicago. So one tradeoff is between locally purchased Internet transit and the combined cost of a transport circuit to Chicago plus Internet transit purchased in Chicago. But additional benefits at a hub like Chicago include the ability to peer with Netflix, Google, Amazon and many others. This can offload 1/2 to 2/3 of the traffic, dramatically reducing the bill for Internet transit.

While the preceding example was a small ISP, the same principals apply at every scale. The goal is to deliver as much of your customers’ traffic directly to the networks with the desired destination addresses and minimize traffic that must transit multiple networks. Larger carriers are typically present in multiple cities and multiple data centers where additional considerations come into effect. The first is “hot-potato routing.” Internet traffic is typically handed off at the first available opportunity. Depending on hand-off points, this could result in one party carrying most cross-country traffic while the other party carries mostly local traffic. To counter this, large carriers’ peering policies typically require connectivity at multiple hubs and relatively balanced traffic flows. The second consideration is economic clout coming from the scale and nature of the operator’s traffic. This shifts over time and among operators leading to peering disputes that have briefly made portions of the Internet unreachable for some users.

4. History and futures

When the NSFnet was turned off in 1995, there were six commercial backbone carriers that did settlement-free peering at four major Internet exchange points. These so-called Tier 1 carriers attempted to form a cartel that did not peer beyond the initial group. However, traffic grew more rapidly than original providers could handle and, by the early 2000s, large groups of secondary carriers were exchanging traffic between each other, effectively forming a donut around the original tier 1 carriers, so that by the mid-2000s the original cartel was irrelevant [3].

The next major evolution began with the emergence of content delivery networks (CDN) like Akamai in the early 2000s. Whether web surfing or buffering video, lower round trip latency improves user experience, so there is big incentive to move content closer to the user. This also reduces the amount of data that must be carried long distances. Larger networks deploy CDN servers within their networks, so most such traffic is entirely locally. Additionally, to reach the broadest number of networks, most CDN providers host CDN servers at major Internet exchanges. By 2010, the majority of inter-domain traffic went to CDNs and Google had emerged as the 2nd largest ISP in the world, by volume [4].

The recent trend is the emergence of multi-lateral peering sites where dozens or hundreds of networks exchange traffic at one location or across one tightly interconnected set of buildings. These peering points arose first in Europe and Asia, where they reduced the amount of local traffic that was routed to the US and back [5]. For example, there are 144 networks interconnected at the Hong Kong Internet Exchange. The group open-ix.org is trying to bring this model to North America. In any event, the focus is on shorter paths and less long haul data transmission as this cuts costs and improves user experience.

5. Peering in practice

A 2011 survey of 4,331 ISP networks (86% of backbone carriers) analyzed 142,210 inter-carrier interconnection agreements with some interesting results [5]. 99.5% were handshake agreements based on commonly understood peering principles, without any written contract, and 99.7% were symmetric. So settlement-free peering dominates. While the majority of networks have less than ten peers, a number of multi-lateral peering agreements are visible in the data since participating operators have hundreds of peers.

Because of the costs in setting up and managing interconnections, there are normally minimum traffic conditions for peering. For example, Google peers at 70+ Internet exchanges and 60+ other facilities around the world. Under their policy, networks at any of these exchanges can peer if they have adequate traffic destined for Google: 100 Mbps for US or EU peering, 25 Mbps for Asia but no minimums at African or South American Internet exchanges [6].

6. Peering Politics

As mentioned above, economic clout plays a role in large carrier interconnection agreements, and occasionally, a breakdown in negotiations results in service disruptions for some users. For example, a November 2010 peering dispute between Level 3 and Comcast arose when Netflix changed CDN partners, from Akamai to Level 3, driving an enormous increase in traffic on Comcast-Level 3 interconnection links. How this was resolved is not public, but the two parties came to some agreement and upgraded links so Comcast customers could again get access to Netflix videos, although nearly three years passed before they announced they had a final agreement [7].

Strong competition in the Internet backbone causes prices to reflect marginal costs, however access networks in the US and many countries are monopolies or duopolies under dramatically less competitive pressure. Recently, major consumer access ISPs like Verizon, Time Warner, Comcast and France Telecom have been using the size of their subscriber bases to demand payments from major content providers like Netflix and Google [8, 9]. How this will play out remains a matter of speculation (and politics).

7. Conclusions

The Internet is a voluntary agreement among network operators to exchange traffic for their mutual benefit [10]. This exchange is remarkable: essentially unregulated, to a great extent informal, not well understood by outsiders, and yet, successfully responding to explosive growth and dramatic technical change for nearly two decades. And, all indications are the system will continue to work for decades to come.

8. References

[1] "Key ICT indicators for developed and developing countries and the world (totals and penetration rates)," International Telecommunications Unions (ITU), Geneva, 27 February 2013