Question about peering

I am curious to know how small ISPs plan peering with other interested parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and say A and C both have open peering policy and assuming the exist in same exchange or nearby. Now at this point is there is any "minimum bandwidth" considerations? Say if A and C have 1Gbps + of flowing traffic - very likely peering would be good idea to save transit costs to B. But if A and C have very low levels - does it still makes sense? Does peering costs anything if ISPs are in same exchange? Does at low traffic level it makes more sense to keep on reaching other ISPs via big transit provider?

Thanks.

--

Anurag Bhatia anuragbhatia.com or simply - http://[2001:470:26:78f::5] if you are on IPv6 connected network!

what does it cost you to peer, versus what does it cost you to not peer?

if you are at the same ix the costs of peering are very low indeed

On Saturday, April 7, 2012, Anurag Bhatia wrote:

> Hello everyone > > > > I am curious to know how small ISPs plan peering with other interested > parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and say > A and C both have open peering policy and assuming the exist in same > exchange or nearby. Now at this point is there is any "minimum bandwidth" > considerations? Say if A and C have 1Gbps + of flowing traffic - very > likely peering would be good idea to save transit costs to B. But if A and > C have very low levels - does it still makes sense? Does peering costs > anything if ISPs are in same exchange? Does at low traffic level it makes > more sense to keep on reaching other ISPs via big transit provider? > > > > Thanks. > > -- > > Anurag Bhatia > anuragbhatia.com > or simply - http://[2001:470:26:78f::5] if you are on IPv6 connected > network! > > Twitter: @anurag_bhatia <https://twitter.com/#!/anurag_bhatia> > Linkedin: http://linkedin.anuragbhatia.com>

Actually, Suresh, I disagree. It depends on the facility/country/continent, the cost of joining the local IX fabric at a reasonable bandwidth, your cost model, and your transit costs. In short, it's not 1999 anymore, and peering is not automatically the right answer from a purely fiscal perspective (though it may be from a technical perspective; see below).

At certain IXes that have a perfect storm of high priced ports and a good assortment of carriers with sufficiently high quality service and aggressive pricing, a good negotiator can fairly easily find himself in a position where the actual cost per megabit of traffic moved on peered bandwidth exceeds the cost of traffic moved on transit _by an order of magnitude_. That's without even factoring in the (low) maintenance cost of having a bunch of BGP sessions around or upgraded routers or whatever.

Sometimes making the AS path as short as possible makes a lot of sense (e.g. when trying to get an anycast network to do the right thing), but assumptions that peering results in lower costs are less true every day.

-r

Suresh Ramasubramanian <ops.lists [at] gmail> writes:

> what does it cost you to peer, versus what does it cost you to not peer? > > if you are at the same ix the costs of peering are very low indeed > > On Saturday, April 7, 2012, Anurag Bhatia wrote: > >> Hello everyone >> >> >> >> I am curious to know how small ISPs plan peering with other interested >> parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and say >> A and C both have open peering policy and assuming the exist in same >> exchange or nearby. Now at this point is there is any "minimum bandwidth" >> considerations? Say if A and C have 1Gbps + of flowing traffic - very >> likely peering would be good idea to save transit costs to B. But if A and >> C have very low levels - does it still makes sense? Does peering costs >> anything if ISPs are in same exchange? Does at low traffic level it makes >> more sense to keep on reaching other ISPs via big transit provider? >> >> >> >> Thanks. >> >> -- >> >> Anurag Bhatia >> anuragbhatia.com >> or simply - http://[2001:470:26:78f::5] if you are on IPv6 connected >> network! >> >> Twitter: @anurag_bhatia <https://twitter.com/#!/anurag_bhatia> >> Linkedin: http://linkedin.anuragbhatia.com>> > > > -- > Suresh Ramasubramanian (ops.lists [at] gmail)

On Sat, Apr 07, 2012 at 06:16:30PM -0400, Robert E. Seastrom wrote: > Sometimes making the AS path as short as possible makes a lot of sense > (e.g. when trying to get an anycast network to do the right thing), > but assumptions that peering results in lower costs are less true > every day.

I keep reading people say that. But wouldn't the same forces that push down the per-megabit cost of transit also push down the per-megabit cost of peering?

> On Sat, Apr 07, 2012 at 06:16:30PM -0400, Robert E. Seastrom wrote: >> Sometimes making the AS path as short as possible makes a lot of sense >> (e.g. when trying to get an anycast network to do the right thing), >> but assumptions that peering results in lower costs are less true >> every day. > > I keep reading people say that. But wouldn't the same forces that push > down the per-megabit cost of transit also push down the per-megabit > cost of peering?

Generally the costs of transit are pushed down by competition. As a vendor your costs for bandwidth/transport/port*bw may drop but you are unlikely to drop your prices to your customers merely because your costs have gone down unless prompted to by a competitor.

In any given IX, cross-connect fibers and peering switch ports are often a monopoly. While not unheard-of for there to be two competing IX switch fabrics available in a single facility, the cross-connects to those competing exchanges are not free, and I'm not aware of any sizeabe facilities that are still "run your own XC and don't pay anyone for it" (of course, as soon as I say that I'll get private email or an IRC message pointing out the corner case).

Consider the case of a peering n00b network (the target of this discussion after all) in hypothetical facility that charges $1000/month for a gigabit ethernet port on the peering fabric. You turn up a connection to this port and discover that (without buying people drinks / sushi dinners / etc at a conference) you can bring up enough peering with other networks to move 150 Mbit/sec on it. That's pretty optimistic for a small player, but still... now you're paying $6.66/mbit for that transit. If you can move 150 Mbit/sec to low-hanging-fruit transit you're probably between 1 and 2gbps total. How's that compare with what you're paying for transit with that level of commit?

On Sat, Apr 07, 2012 at 07:25:24PM -0400, Robert E. Seastrom wrote: > Generally the costs of transit are pushed down by competition. As a > vendor your costs for bandwidth/transport/port*bw may drop but you are > unlikely to drop your prices to your customers merely because your > costs have gone down unless prompted to by a competitor.

ah, so it's not the cost of production that is the problem, it is the 'natural monopoly' state of an IX that is the problem.

It seems like that problem could be overcome by making the IX a cooperative owned by the members, maybe?

> Consider the case of a peering n00b network (the target of this > discussion after all) in hypothetical facility that charges > $1000/month for a gigabit ethernet port on the peering fabric. You

I am in almost that exact position (A peering n00b network) - Of couse, I'm fairly certain I'm paying sucker prices, but I can get a gigE to any2 at 55 s market for less than a third the price you quote.

>It seems like that problem could be overcome by making the >IX a cooperative owned by the members, maybe?

Even if an IX is a cooperative, that doesn't say anything about their costs and the costs of interconnection. Networks and buildings and cross-connects can get cheap for lots of reasons, but the nature of the ownership isn't really a factor. Cooperatives can be as poorly run or have as high costs as any commercial facility.

In fact, you could argue that without some cross-subsidy of co-lo or one of the providers 'donating' space, a small cooperative is likely to be more expensive to put together than a large colo facility that has lots of revenue streams. Or you could argue the opposite. I'm just pointing out that motivation and ownership don't necessarily dictate final costs.

That being said...

>I am in almost that exact position (A peering n00b network) - Of >couse, I'm fairly certain I'm paying sucker prices, but I can get a >gigE to any2 at 55 s market for less than a third the price you quote.

Well, bully for you, but at this very instant I'm looking at a contract from PCCW which has a component of a cross-connect in Telecity London (Harbour Exchange) where the cross-connect has been priced out at USD 2400/month (maybe that also includes 1U of space; it's hard to tell).

I do understand that this is NANOG with emphasis on the "NA" part, and so costs in other geographies may not be all that interesting, but some facilities do charge an arm and a leg (or maybe PCCW is screwing us over on the proposal).

> On Sat, Apr 07, 2012 at 07:25:24PM -0400, Robert E. Seastrom wrote: >> Generally the costs of transit are pushed down by competition. As a >> vendor your costs for bandwidth/transport/port*bw may drop but you are >> unlikely to drop your prices to your customers merely because your >> costs have gone down unless prompted to by a competitor. > > ah, so it's not the cost of production that is the problem, it is > the 'natural monopoly' state of an IX that is the problem. > > It seems like that problem could be overcome by making the > IX a cooperative owned by the members, maybe?

The whole datacenter?

>> Consider the case of a peering n00b network (the target of this >> discussion after all) in hypothetical facility that charges >> $1000/month for a gigabit ethernet port on the peering fabric. You > > I am in almost that exact position (A peering n00b network) - Of > couse, I'm fairly certain I'm paying sucker prices, but I can get a > gigE to any2 at 55 s market for less than a third the price you quote. > > just a data point.

> > Actually, Suresh, I disagree. It depends on the > facility/country/continent, the cost of joining the local IX fabric at > a reasonable bandwidth, your cost model, and your transit costs. In > short, it's not 1999 anymore, and peering is not automatically the > right answer from a purely fiscal perspective (though it may be from a > technical perspective; see below). > > At certain IXes that have a perfect storm of high priced ports and a > good assortment of carriers with sufficiently high quality service and > aggressive pricing, a good negotiator can fairly easily find himself > in a position where the actual cost per megabit of traffic moved on > peered bandwidth exceeds the cost of traffic moved on transit _by an > order of magnitude_. That's without even factoring in the (low) > maintenance cost of having a bunch of BGP sessions around or upgraded > routers or whatever. > > Sometimes making the AS path as short as possible makes a lot of sense > (e.g. when trying to get an anycast network to do the right thing), > but assumptions that peering results in lower costs are less true > every day. > > -r > > Suresh Ramasubramanian <ops.lists [at] gmail> writes: > >> what does it cost you to peer, versus what does it cost you to not peer? >> >> if you are at the same ix the costs of peering are very low indeed >> >> On Saturday, April 7, 2012, Anurag Bhatia wrote: >> >>> Hello everyone >>> >>> >>> >>> I am curious to know how small ISPs plan peering with other interested >>> parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and say >>> A and C both have open peering policy and assuming the exist in same >>> exchange or nearby. Now at this point is there is any "minimum bandwidth" >>> considerations? Say if A and C have 1Gbps + of flowing traffic - very >>> likely peering would be good idea to save transit costs to B. But if A and >>> C have very low levels - does it still makes sense? Does peering costs >>> anything if ISPs are in same exchange? Does at low traffic level it makes >>> more sense to keep on reaching other ISPs via big transit provider? >>> >>> >>> >>> Thanks. >>> >>> -- >>> >>> Anurag Bhatia >>> anuragbhatia.com >>> or simply - http://[2001:470:26:78f::5] if you are on IPv6 connected >>> network! >>> >>> Twitter: @anurag_bhatia <https://twitter.com/#!/anurag_bhatia> >>> Linkedin: http://linkedin.anuragbhatia.com>>> >> >> >> -- >> Suresh Ramasubramanian (ops.lists [at] gmail)

On 4/6/2012 3:11 PM, Anurag Bhatia wrote: > I am curious to know how small ISPs plan peering with other interested > parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and say > A and C both have open peering policy and assuming the exist in same > exchange or nearby. Now at this point is there is any "minimum bandwidth" > considerations? Say if A and C have 1Gbps + of flowing traffic - very > likely peering would be good idea to save transit costs to B. But if A and > C have very low levels - does it still makes sense? Does peering costs > anything if ISPs are in same exchange? Does at low traffic level it makes > more sense to keep on reaching other ISPs via big transit provider?

One thing to consider is that peering can benefit both networks not just because of bandwidth savings, but because (given sufficient clue) they can deliver better performance and reliability to their mutual customers.

Most of the time no. ISP A and ISP C probably don't have alot of traffic destined for each other's AS's. Without other peers in an IX sort of model the link would probably be mostly devoid of (useful) traffic. Although, if ISP A and C were small regional ISP's and they could get free peering from someone like netflix that may be worth while, but I digress. Another interesting occurrence would be if ISP A shifted it's metrics to force it's transit traffic into ISP C's AS offloading the cost of the eventual ISP B hop to ISP C. (assuming someone announces the full table) I've also seen ISP A re-announce ISP C's routes to their upstreams with preferred metrics in order to make the link "one-sided" and begin billing ISP A.

2012/4/6 Anurag Bhatia <me [at] anuragbhatia>

> Hello everyone > > > > I am curious to know how small ISPs plan peering with other interested > parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and say > A and C both have open peering policy and assuming the exist in same > exchange or nearby. Now at this point is there is any "minimum bandwidth" > considerations? Say if A and C have 1Gbps + of flowing traffic - very > likely peering would be good idea to save transit costs to B. But if A and > C have very low levels - does it still makes sense? Does peering costs > anything if ISPs are in same exchange? Does at low traffic level it makes > more sense to keep on reaching other ISPs via big transit provider? > > > > Thanks. > > -- > > Anurag Bhatia > anuragbhatia.com > or simply - http://[2001:470:26:78f::5] if you are on IPv6 connected > network! > > Twitter: @anurag_bhatia <https://twitter.com/#!/anurag_bhatia> > Linkedin: http://linkedin.anuragbhatia.com> >

>On 4/6/2012 3:11 PM, Anurag Bhatia wrote: >> I am curious to know how small ISPs plan peering with other interested >> parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and >>say >> A and C both have open peering policy and assuming the exist in same >> exchange or nearby. Now at this point is there is any "minimum >>bandwidth" >> considerations? Say if A and C have 1Gbps + of flowing traffic - very >> likely peering would be good idea to save transit costs to B. But if A >>and >> C have very low levels - does it still makes sense? Does peering costs >> anything if ISPs are in same exchange? Does at low traffic level it >>makes >> more sense to keep on reaching other ISPs via big transit provider? > >One thing to consider is that peering can benefit both networks not just >because of bandwidth savings, but because (given sufficient clue) they >can deliver better performance and reliability to their mutual customers. > >

At the previous regional ISP i worked for we peered with google, facebook, yahoo, pandora and several other content providers at Any2 exchange (coresite). We capitalized on that link since a tremendous amount of our traffic was destined for those networks.

The cost to join that exchange was relatively cheap compared to what we were paying for transit. You may want to look for a similar exchange at your pop.

> We are cross-connected with several ISPs at a couple of data > centers. > Very helpful in one situation as several of us share a soft-switch. > > Justin > > -----Original Message----- > From: Rob Szarka <szlists [at] szarka> > Date: Wednesday, May 9, 2012 4:50 PM > To: <nanog [at] nanog> > Subject: Re: Question about peering > > >On 4/6/2012 3:11 PM, Anurag Bhatia wrote: > >> I am curious to know how small ISPs plan peering with other interested > >> parties. E.g if ISP A is connected to ISP C via big backbone ISP B, and > >>say > >> A and C both have open peering policy and assuming the exist in same > >> exchange or nearby. Now at this point is there is any "minimum > >>bandwidth" > >> considerations? Say if A and C have 1Gbps + of flowing traffic - very > >> likely peering would be good idea to save transit costs to B. But if A > >>and > >> C have very low levels - does it still makes sense? Does peering costs > >> anything if ISPs are in same exchange? Does at low traffic level it > >>makes > >> more sense to keep on reaching other ISPs via big transit provider? > > > >One thing to consider is that peering can benefit both networks not just > >because of bandwidth savings, but because (given sufficient clue) they > >can deliver better performance and reliability to their mutual customers. > > > > > > > >