Lee Hutchinson has a problem. My fellow Ars writer is a man who loves to watch YouTube videos—mostly space rocket launches and gun demonstrations, I assume—but he never knows when his home Internet service will let him do so.

"For at least the past year, I've suffered from ridiculously awful YouTube speeds," Hutchinson tells me. "Ads load quickly—there's never anything wrong with the ads!—but during peak times, HD videos have been almost universally unwatchable. I've found myself having to reduce the quality down to 480p and sometimes even down to 240p to watch things without buffering. More recently, videos would start to play and buffer without issue, then simply stop buffering at some point between a third and two-thirds in. When the playhead hit the end of the buffer—which might be at 1:30 of a six-minute video—the video would hang for several seconds, then simply end. The video's total time would change from six minutes to 1:30 minutes and I'd be presented with the standard 'related videos' view that you see when a video is over."

Hutchinson, a Houston resident who pays Comcast for 16Mbps business-class cable, is far from alone. As one Ars reader recently complained, "YouTube is almost unusable on my [Verizon] FiOS connection during peak hours." Another reader responded, "To be fair, it's unusable with almost any ISP." Hutchinson's YouTube playback has actually gotten better in recent weeks. But complaints about streaming video services—notably YouTube and Netflix—are repeated again and again in articles and support forums across the Internet.

Why does online video have such problems? People may assume there are perfectly innocent causes related to their computers or to the mysterious workings of the Internet. Often, they're correct.

But cynical types who suspect their Internet Service Providers (ISPs) intentionally degrade streaming video may be right as well. No, your ISP (probably) isn't sniffing your traffic every time you click a YouTube or Netflix link, ready to throttle your bandwidth. But behind the scenes, in negotiations that almost never become public, the world's biggest Internet providers and video services argue over how much one network should pay to connect to another. When these negotiations fail, users suffer. In other words, bad video performance is often caused not just by technology problems but also by business decisions made by the companies that control the Internet.

Please wait. Keep waiting.

These business decisions involve "peering" agreements that Internet companies make to pass traffic from one to another and negotiations over caching services that store videos closer to people's homes so they can load faster in your browser. When Internet providers refuse to upgrade peering connections, traffic gets congested. When ISPs refuse to use the caching services offered by the likes of Google and Netflix, video has to travel farther across the Internet to get to its final destination—your living room.

The negotiations can lead to brinksmanship and bad blood. Recent public examples of such spats include:

November 2010: After Internet backbone provider Level 3 signs a deal with Netflix to distribute video, Comcast demands money from Level 3 for carrying traffic over the proverbial "last mile" to Comcast subscribers.

January 2011: European ISPs Deutsche Telekom, Orange (formerly France Telecom), Telecom Italia, and Telefónica commission a report saying companies like Netflix and Google's YouTube service should give ISPs a lot more money.

August 2011: Cogent, another Internet backbone provider that handles Netflix traffic, files a complaint in France against Orange, saying the ISP is providing inadequate connection speeds.

January 2013: Free, a French ISP, is accused of slowing down YouTube traffic by failing to upgrade infrastructure (but is later cleared of intentionally degrading YouTube traffic by the French regulator). Free also temporarily blocks ads on YouTube and other video services by sending an update to its modems.

January 2013: Orange and Google have a similar dispute, with Orange CEO Stephane Richard claiming victory. He says that Google is paying Orange to compensate the operator for mobile traffic sent from Google servers.

January 2013: Time Warner refuses Netflix's offer of a free caching service that would provide better performance to Netflix users on Time Warner's network.

June 2013: Cogent accuses Verizon of allowing "ports" between the two providers to fill up, degrading Netflix performance for Verizon customers.

July 2013: The European Commission opens an antitrust probe into whether ISPs abused market positions in negotiations with content providers, and it searches the offices of Orange, Deutsche Telekom, and Telefónica. Separately, the French government demands details of interconnection agreements involving AT&T and Verizon.

In the most extreme cases, large Internet companies stop passing traffic to one another entirely. (This happened in 2005 with France Telecom and Cogent, in 2005 with Cogent and Level 3, and in 2008 with Sprint and Cogent.) But recent disputes have been less likely to lead to a complete severing of ties. "That type of reaction to a policy is becoming less common, possibly because it's so easy to publicize it," Reggie Forster, director of network engineering at XO Communications, told Ars. "They tend to want to keep that quiet."

Instead, network operators can degrade traffic by failing to upgrade connections without severing them entirely. The public won't realize that's what's going on unless negotiations become so contentious that one party makes them public—or a government decides to investigate.

Degraded connections disproportionately affect the quality of streaming video because video requires far more bits than most other types of traffic. Netflix and YouTube alone account for nearly half of all Internet traffic to homes in North America during peak hours, according to research by SandVine. And customers are far more likely to be annoyed by a video that stutters and stops than by a webpage taking a few extra seconds to load.

To get to the root of these problems, we need to take a step back and talk about the Internet itself. The very name "Internet" suggests many networks interconnecting, but few people know how the terms of these connections are negotiated. Understanding the business relationships that allow the Internet to exist in its present form is crucial to understanding the subtle and not-so-subtle ways Internet companies can bring your YouTube and Netflix videos to a slow stutter.

240 Reader Comments

I've not been pleased at all with the offerings in my area. Insight BB was a decent company, imo. Then they sold out to TWC. Now my modem doesn't keep a connection during peak hours. ("Power down, power up!")

I've not had too much problem streaming video. So my area must be lucky. Having stated that, this line struck me:

Quote:

Verizon has said again and again that it could upgrade to gigabit; it just doesn't think that consumers really want it.

I fall into that group. With my 10Mb connection, I and my family are able to do all their streaming, browsing, and gaming it wants to do. They keep offering me to upgrade to their 20.0, 30.0 or whatever speed. I just haven't seen the need, especially since the upgrade would cost significantly more.

And this article kind of makes me think that upgrading my connection wouldn't do any good anyway. Whether I get 1Tb of throughput to that last mile, it doesn't really matter if Netflix, Microsoft, Ubuntu (or other Linux OS's), Youtube, etc, etc, can't get that speed through my ISP's transit/peer connections.

My internet provider (laughingly called that) is the justly vilified Comcast. Their service is horrid, their prices lamentable and they still strive to sell us cable and phone while those were the final straws that broke my last bit of patience with their company. They have an effective monopoly here because Verizon and ATT both reneged on their promises of fiber to the home.

Comcast made no such promise but they did insist on my paying a maintenance fee for the modem that I paid for. When confronted they Customer Insulting Service constantly commented on how the customer was very important to Comcast even though they believed that renting me my own equipment was a proper course of action. When they finally realized that yes one is not able to rent someone else equipment back to them there were quick to remove the charge. But they had to be told why first. Ignorance runs rampant through that organization following on the heels of a deep seated greed.

A horrible company in all aspects of the concept.

A similar story with me and Comcast here in Atlanta - and it got even stupider: after acknowledging the mistake of charging a modem rental fee, suddenly so - after about 8 years of owning my modem, somebody in the chain of idiots admitted there also had been a few dollar charge for the "benefit" of them accessing my records to make the "change." I hate them with a passion but my speeds are amazing with little or no buffering issues, even on 1080p. My only alternative is Uverse, with a hard sell I just don't trust - the only company I hate more than Comcast is ATT. Probably the other way around now but I feel like I'm trapped.

ISP subscribers PAY for bandwidth. ISPs are violating his contract to the client either if they charged for a service that they cannot provide, or if they refuses to provide a service who his subscribers PAID.

I say it again: the subscriber PAYS for the bandwidth, so the ISP has no right to violate his obligation, and he doesn’t need extra money from Google or anybody. That’s like cable companies pretending to charge content makers for using his signal at profit, meanwhile taking money from his own clients.

It's beyond ridiculous how often misplaced greed holds us back. Where they are seeing a chance to scam money off from Youtube: I am seeing an opportunity to sell network equipment designers rack space for real world testing. Many times over our species would have been doomed if it were not for cooperation, why are they so damned intent on being competitive when the alternative is beyond a shadow of a doubt far more profitable.

Bit torrent impacting other users is all down to concurrency/contention, the more users on a card, the more the sockets and bandwidth has to be split. Contention of 1-24 isnt too bad, but Ive seen as bad as 1-50 in some situations, with business class maybe being 1-12 or 1-16 - note contention rates vary depending on who you talk to, where theyre measured and who's trying to make theirs look better.

...

So we can buy enough bandwidth to ensure our customers get what they pay for, or at least enough to keep 'average' use ticking over. Fair enough, everyone has to buy the bandwidth if youre not a backbone or provider, thing is, we can buy it all day long, but we're still dependant on existing infrastructure to deliver it - infrastructure owned and maintained by the two telcos. Sure we have our own auth / radius servers, but the pstn, exchange, infrastructure its all the telcos, we just get local loop access.

There's the problem, the telco's aren't offering you guys decent infrastructure so you have to rate limit. Typically the ISP owned the infrastructure, but you're special.

It sounds like you're doing the best you can with what you got.

The problem most people have a large incumbent ISPs who have the funds to upgrade their infrastructure, but refuse to because they have no competition.

Once you have infrastructure that doesn't have issues with contention, then you just keep your trunk upgraded to handle the load.

Arg, when the government decided to give cable and telephone companies oligopolies, I don't think anyone envisioned them basically acting like trolls, trying to collect tolls from any network traffic that passes through to their customers who they supposedly have the best interest in heart for.

Hutchinson, a Houston resident who pays Comcast for 16Mbps business-class cable, is far from alone.

I have 27Mbps (which measures at 35) Comcast business-class cable out of Boston. It is by far the best ISP I have had in my 19 years of having a static IP. There is always one Netflix stream coming in, often up to 3 (or 4 if my son and his iPad are here). I use youtube as my ipod, it runs 12 hours a day. And we have absolutely 0 complaints.

Actually, that's not quite true. Comcast will not allow me the privilege of giving them an additional $80/mo for TV channels because business don't get TV and they can't handle two accounts from the same address. So I have no options to watch Red Sox. Go figure.

I think there are two different issues at work here, and this article only covered one of them. Personally, when I'm at home I rarely have bandwidth problems with youtube, although on a 1080p video it's close. My issue is the fact something changed ~1.5 years ago? (not sure exactly) on google's/youtube's end, where videos switched from buffering/caching the whole file as soon as possible, to only getting a small piece of it no matter how long you wait. Heck, you used to be able to buffer a whole video on a dial up connection if you left it to itself long enough. You also used to be able to move around in the downloaded area (go back to beginning etc.) while the remainder was downloading, but now moving the cursor manually resets the whole buffered file. From everything I can tell, this is a separate issue from the one discussed in the article, and seemingly a policy change (now it may have been a policy changed based on some peering stuff, I have no idea). I was really hoping to get some solid info on this issue as I haven't been able to find much on it. Maybe a followup article?

That was done when they started monetizing the content.

If they allow a full buffer, its stored, cryptically (i.e. name and extension not easily identifiable) on the hard drive until you left the page when it was purged and a simple search by date and an educated guess based on size revealed the file as long as the page was still open; which you could then copy and rename to, practically, anything with any extension VLC or a codec pack decoded and play it whenever.. As long as it was a movie file and had an extension recognized as video, you did not need to get the extension right...

I used to do it all the time and FTR, YouTube is not the only streaming service that allows only limited ram buffering. (if they keep it small enough they assume it stays in ram, and if not you only get a fragment.)

I work for an isp, theres more going on under the hood than you might think.

Bit torrent users can cause more than a small amount of hassles, because many of htem dont lock down connections/sockets and end up saturating their connections with scores of 1-2k connections. The impact on the overall network is startling in its degredation, so to protect the majority of customers we packet shape torrent traffic to low levels during 8am to midnight. If we didnt, the torrenters will not only eat all available (and purchased) bandwidth, but it causes latency and overhead for our other customers..

Let's be blunt here: the only reason why you have to throttle is because you oversubscribed your capacity, in short selling more than you can actually deliver. And instead of investing in expanding your backbone you prefer to keep the money and use throtteling to compensate for the lack of investment.

Quote:

Nickel and diming, hardly, just trying to keep a service going.

It's exactly that, Nickel and diming, with the intention to squeeze as much money out as possible while avoiding to invest in a proper infrastructure. So no surprise that you can't cope when customers actually make use of what they are paying for.

You do realize that ISPs work on the assumption that 95+% of connections are going to be effectively idle at any given time? If they provisioned the networks on the assumption that you're going to be using your full bandwith all the time, you would be paying $100+ per 1mbps. The ONLY reason they can give you 40-100mbps connections today is because of that assumption your neighbors are only using a tiny fraction of their network.

If they offered you 10 megabits and they provisioned for every single user to have 10 megabits, they would have to lay out OC48 lines every 248 customers. That's at least one per neighborhood with some neighborhoods needing multiple. That cost would be through the roof and just for 10 meg service. Not to mention the crazy upgrades to their backbones, routers and datacenters.

So instead, they estimate. They figure an area has 500 users, 95% are using less than 1mbps on average and 5% are using the max. If the area has say 1000mbps connectivity all together they figure they can offer everyone 20 meg service without capping out. Unfortunately that 95% tends to fluctuate, so there are certain periods when those 5% get less than the max because the 95% is going over expected usage. If they didn't use such estimations, everyone would get 2meg max, all day.

Now that's not to say it doesn't happen where an area NEEDS an upgrade and the company fights it. That would be naive. But the flat argument of "You offer X speed, I should get that exact speed ALL THE TIME NO MATTER WHAT" winds up fucking over everyone in the end. If you want a guaranteed bandwith go to business class and see how expensive that stuff gets.

You do realize that ISPs work on the assumption that 95+% of connections are going to be effectively idle at any given time? If they provisioned the networks on the assumption that you're going to be using your full bandwith all the time, you would be paying $100+ per 1mbps.

No, I did not realize this because its not in the TOS...

Actually, I do knowthis (well, I dispute the final cost and I'll get into later why its par for the course for you) but not because any ISP was honest about it and the "any given time" part is intentional obfuscation on your part.... you know full well, as do I, as do the ISPs when peak usage is, so the "any given time" snippet just a goosestep lying.

and, you reveal much about yourself when you imply ignorance is the fault of the ignorant and thus, justifiably, fair game.

I think there are two different issues at work here, and this article only covered one of them. Personally, when I'm at home I rarely have bandwidth problems with youtube, although on a 1080p video it's close. My issue is the fact something changed ~1.5 years ago? (not sure exactly) on google's/youtube's end, where videos switched from buffering/caching the whole file as soon as possible, to only getting a small piece of it no matter how long you wait. Heck, you used to be able to buffer a whole video on a dial up connection if you left it to itself long enough. You also used to be able to move around in the downloaded area (go back to beginning etc.) while the remainder was downloading, but now moving the cursor manually resets the whole buffered file. From everything I can tell, this is a separate issue from the one discussed in the article, and seemingly a policy change (now it may have been a policy changed based on some peering stuff, I have no idea). I was really hoping to get some solid info on this issue as I haven't been able to find much on it. Maybe a followup article?

My guess here was that this move was designed to disrupt software that could download entire videos from youtube.... i.e. there is no way with this that you can just download the entire file from the server as fast as you can get it. Obviously there are ways to download entire videos still but it will take longer and therefore be a little trickier to automate.

If they provisioned the networks on the assumption that you're going to be using your full bandwith all the time, you would be paying $100+ per 1mbps.

Actually, dedicated bandwidth is running less than $1/mbit from a Tier 1, it's due to be about $0.40/mbit by 2014 and even cheaper if your ISP has peering, like with Google or Netflix, as those two make up about 50% of their peak bandwidth.

A customer uploading 100Mb/s all month long will cost about $40 in trunk bandwidth for an ISP by next year some time. Add another $20 for infrastructure costs and double the price for random other costs, and you're talking about $120/m for a single customer uploading 100Mb/s. By 2016, prices should be around $0.20/mbit. Prices drop 50% year to year, so could be as low as $0.10/mbit.

When you see prices for dedicated lines, your not paying $100/mbit for bandwidth, you're paying $99/mbit for an SLA and a dedicated link, the $1 is your bandwidth costs.

Another fun fact is that most Tier 1s charge based on 95% percentile of bandwidth, so if you transfer 100Mb/s for 1.2 hours per day or 100Mb/s for 24 hours per day, you pay the same.

Because 95% percentile ignores bursts, you could transfer 50Mb/s with bursts up to 100Mb/s and still only pay for 50Mb/s, so long as those bursts last shorter than 1.2hr/day. I really depends on your agreement.

If they offered you 10 megabits and they provisioned for every single user to have 10 megabits, they would have to lay out OC48 lines every 248 customers. That's at least one per neighborhood with some neighborhoods needing multiple. That cost would be through the roof and just for 10 meg service. Not to mention the crazy upgrades to their backbones, routers and datacenters.

I should have had this in my last reply, but was tired and missed this part.

For large networks, you don't need dedicated trunk bandwidth, just dedicated local bandwidth to the nearest aggregation point. The largest Internet Exchanges(IX) in the world actually have very predictable loads. Transit loads vary very little day-to-day, like in the 100Mb/s ranges.

One could claim to give an entire city dedicated bandwidth for 1Gb/s, but given a large enough population, you could predict almost exactly how much bandwidth will be used and not have to worry about burst causing congestion.

You should see some of the core routers out there. 100Gb/s per port with 4 ports per slot and support up to 96 slots and all non-blocking layer3 routing. 38Tb/s of routing speed, but the busiest IX only see about 2.5Tb/s.

The entire Internet is around 100Tb/s, which can be handled by 3 of these routers, except the Internet is spread all around the world across hundreds of thousands of routers, so no worries.

Newer chassis being sold have 400-600 1Gb Ethernet ports and 400-600Gb of uplink. Then you can plug each of these 100Gb ports into the above router, but why would you? The busiest IX that has most of Europe moving through it, is only 2.5Tb/s. A small city could easily have full non-blocking internal bandwidth, but given averages, the trunk bandwidth would only need to be a small fraction.

So yes, you could give everyone in any major city in the world, "dedicated bandwidth" and just use the laws of large numbers to make sure you can keep your trunk well supplied.

So if you go to a datacenter to put up your companywebsite, do you get free bandwidth because your ISP and everyone else is paying for it... no. That is my point. Everyone else gets billed until you are "too big to bill". If you actually had to pay the bills you'd know where I'm coming from.

If I go to a data center or hosting provider and put up a server, then yes I pay for my bandwidth. As in the bandwidth my server uses. I pay my hosting provider or the data center operator. And I've paid, period full stop. I haven't paid part, I've paid it all.

So now an ISP comes along and wants me to pay for someone else's bandwidth. In this case, their customer's bandwidth. Why? I've paid for my bandwidth. If their customer wants to use bandwidth, let their customer pay for their bandwidth just like I did. If the ISP doesn't want to bill their customer for their bandwidth usage, how is that my problem?

And as far as the bandwidth between the ISP and the data center, well, that's an issue for the contract between the ISP and the data center. Let them sort out who pays who how much under their contract. The real issue for the ISPs is that they're the demand side, their customers want the data from the servers in the data center, and as usual the demand side pays the supply side and the supply side delivers the goods. Seems fair to me. Let the ISPs bill their customers to cover what the ISPs have to pay for what their customers want. Or start telling their customers they can't have that.

So I'm a bit late to the party here. But read my post history, I've posted extensively about this issue before and I've dealt with it professionally pretty extensively as well.

From the telco point of view (which I'm not passing judgement on here) there *are* two networks:

1. There is the national network, which the telco operates and connects every city that it has operations in. This network is run over fiber-optic lines and has interconnects out the wazoo. This network is pretty cheap to run, but needs to be incredibly reliable. Six-sigma reliable, which is kind of expensive. So they feel justified in charging net traffic providers to access this network (but more on that later.) If you look at it, the amount of money the telcos make from this kind of activity is insignificant. They basically use it to fund network upgrades.

2. There is the "last mile" network. This is what connects your house to their datacenter. This is *incredibly*, *incredibly* expensive to upgrade and maintain. Just looking at Comcast (the largest ISP in the US) their capital expenditure comes out to around $4.9 billion; most of which is spent on network upgrades. Every year. Forever. This is what your monthly cable bill goes to (well, that and content licensing fees; many of which are subsidized by HSD subscribers; but that's another matter...)

None of this is inherently bad to the consumer. However, the most damaging aspect of this arrangement comes in when you realize that, according to convention, carriers *get paid* when accepting traffic from a network. This means that, in order to maintain their balance of power and stay on the side of companies who get paid to accept traffic, most consumer ISPs have an active incentive to ensure connections remain asynchronous and weighted toward downstream bandwidth. Services that use an inordinate amount of upstream bandwidth like Dropbox, Bittorrent and Tor are discouraged and traffic shaped accordingly.

It's my job to handle this kind of crap, and you know what? I hate it. I hate it because it sucks for consumers, but at the same time, it can't be any other way. Not because these companies involved are evil, but because when you think about it as an end-to-end service, someone has to pay to upgrade the network. There are three options as to who gets to pay to upgrade the network to handle all this additional traffic from these services that consumers are using; tell me which one makes the most sense:

1. You, the consumer. Well, I mean, of course you're going to pay more, but that'll only really cover the last-mile upgrades.2. The cable/telco company. I mean, sure. This sounds fair. But it takes effort and most importantly, *money* to keep improving and maintaining the last-mile network.3. The company providing the video that you're watching. That you are either paying for or watching ads that they get paid to deliver to you.

In traditional media setups, 2 and 3 above were the same, so yeah, it's fair that the cable company pays more. But anymore, 2 and 3 are different companies. Netflix would have you believe that the choice is between 1 and 2. The cable companies want you to believe the choice is between 2 and 3. But really, the choice is between 1 and 3, because you're going to pay what you're going to pay to get to the internet, and Netflix could afford their interconnect fees if they took less profit and got fucked like everyone else paying for content rights has over the last 20 years.

So if you go to a datacenter to put up your companywebsite, do you get free bandwidth because your ISP and everyone else is paying for it... no. That is my point. Everyone else gets billed until you are "too big to bill". If you actually had to pay the bills you'd know where I'm coming from.

If I go to a data center or hosting provider and put up a server, then yes I pay for my bandwidth. As in the bandwidth my server uses. I pay my hosting provider or the data center operator. And I've paid, period full stop. I haven't paid part, I've paid it all.

So now an ISP comes along and wants me to pay for someone else's bandwidth. In this case, their customer's bandwidth. Why? I've paid for my bandwidth. If their customer wants to use bandwidth, let their customer pay for their bandwidth just like I did. If the ISP doesn't want to bill their customer for their bandwidth usage, how is that my problem?

And as far as the bandwidth between the ISP and the data center, well, that's an issue for the contract between the ISP and the data center. Let them sort out who pays who how much under their contract. The real issue for the ISPs is that they're the demand side, their customers want the data from the servers in the data center, and as usual the demand side pays the supply side and the supply side delivers the goods. Seems fair to me. Let the ISPs bill their customers to cover what the ISPs have to pay for what their customers want. Or start telling their customers they can't have that.

Except you have demand and supply mixed up. See, the "demand" on the internet is for packet sinks; not for content. What's in demand is a high-speed uplink to upload your content to. Storage is cheap, I can download everything you send me and throw it on a storage array. But how do I get rid of the stuff on my array if I'm supposed to pass it along? I need you to help me empty my warehouse so I can store more stuff.

Hence why packets are not in demand, *packet sinks* are in demand. If you think about it that way, the market makes more sense.

Using Verizon's logic, they should be paying me each month. They're sending MUCH more data to my [home] network each month than I am sending to them.

Last mile network. Peering rules don't apply when you are required by the government to provide and maintain connectivity to all potential customers in an area (even if they don't use it). In these areas, the fixed costs of maintaining the line outweigh any marginal revenue gained from operating the line in the absence of subscriber fees.

If they provisioned the networks on the assumption that you're going to be using your full bandwith all the time, you would be paying $100+ per 1mbps.

Actually, dedicated bandwidth is running less than $1/mbit from a Tier 1, it's due to be about $0.40/mbit by 2014 and even cheaper if your ISP has peering, like with Google or Netflix, as those two make up about 50% of their peak bandwidth.

A customer uploading 100Mb/s all month long will cost about $40 in trunk bandwidth for an ISP by next year some time. Add another $20 for infrastructure costs and double the price for random other costs, and you're talking about $120/m for a single customer uploading 100Mb/s. By 2016, prices should be around $0.20/mbit. Prices drop 50% year to year, so could be as low as $0.10/mbit.

When you see prices for dedicated lines, your not paying $100/mbit for bandwidth, you're paying $99/mbit for an SLA and a dedicated link, the $1 is your bandwidth costs.

Another fun fact is that most Tier 1s charge based on 95% percentile of bandwidth, so if you transfer 100Mb/s for 1.2 hours per day or 100Mb/s for 24 hours per day, you pay the same.

Because 95% percentile ignores bursts, you could transfer 50Mb/s with bursts up to 100Mb/s and still only pay for 50Mb/s, so long as those bursts last shorter than 1.2hr/day. I really depends on your agreement.

This is all correct, I wholeheartedly agree. But you're talking about from a Tier 1, not last mile providers. Exelius said it perfectly "2. There is the "last mile" network. This is what connects your house to their data center. This is *incredibly*, *incredibly* expensive to upgrade and maintain."

While the Tier 1's have vastly advanced networks, their topologies are immensely simpler than last mile providers. The last mile providers will be running thousands of switches and routers and terminating in tens of thousands of locations for every one run by a tier 1.

Yes given a large enough demographic you can gauge spikes and usage easier. The problem is for the local providers they provision based on no more than a few miles at a time. Those same lines and equipment are often expected to last up to 10 years (sometimes more!) which makes it even harder to judge.

And in the end, it doesn't matter how much the last mile providers provision, they will ALWAYS oversell their network, especially residential. Because at any given time, the vast majority of connectionss will be sitting idle or being under utilized resulting in excess bandwith. Economically it makes no sense to let a resource sit idle, good business sense says you sell that to someone. Even if they provision enough for everyone to have 100 megs per second all the time, they will continue to sell folks a 200 meg service because someone will pay for it and because they're going to have the bandwith to spare.

This isn't to say they aren't greedy, highly profitable bastards. There are a lot of places they need to improve their service. But simply that the sales methodology isn't really to blame. Like I said, everything you said is completely true, it just doesn't apply the same to last mile, which provides service to all but a select few areas and businesses.

You do realize that ISPs work on the assumption that 95+% of connections are going to be effectively idle at any given time? If they provisioned the networks on the assumption that you're going to be using your full bandwith all the time, you would be paying $100+ per 1mbps.

No, I did not realize this because its not in the TOS...

Actually, I do knowthis (well, I dispute the final cost and I'll get into later why its par for the course for you) but not because any ISP was honest about it and the "any given time" part is intentional obfuscation on your part.... you know full well, as do I, as do the ISPs when peak usage is, so the "any given time" snippet just a goosestep lying.

and, you reveal much about yourself when you imply ignorance is the fault of the ignorant and thus, justifiably, fair game.

I pity all around you.

I'd say you're the ignorant one here. Double check your TOS, if you have residential service I guarantee you have a line in similar to this "The speed of the Service will vary based on the network or Internet congestion, your computer configuration, the condition of your telephone line and the wiring inside your location, among other factors. We make no guarantees or representations related to the download or upload speeds."

And that's from the vaunted "Dedicated" line of DSL. Simply there's a reason they love to include the words "Up to"

To be honest I really hate Youtube since they stopped buffering when a video is paused.

At least with slow connections, in the past, you could watch videos in high quality if you wanted, just hit play, then pause and let it load completely.

This is not possible with Youtube anymore. It will stop loading if the video is paused. I understand this is to save them money on bandwidth but it means if you have a slow connections you are out of luck.

Most streaming providers copied this horrendous feature and only buffer to a certain point when the playback is paused. It seems they did not realized that some people will actually pause on purpose, exactly to avoid that, buffering issues.

You know, this idea of just having everyone eat the cost of sending Netflix's data is great when it's netflix, because I'm a netflix customer and it's awesome, but when we let everyone go out of ratio on their settlement-free peers because they're ESPN's isp, or the christian streaming media blah blah blah, or whatever else you can imagine getting big and not wanting to participate in, it's going to suck. That's just turning in to cable TV all over again, with no a la carte pricing.

We pay netflix and netflix pays Cogent, sure; we pay our own ISPs so we can download (or upload) stuff, sure, but we're not paying our own ISPS to make sure that every content provider on the internet has enough bandwidth to send to everyone else. Basing settlement free peering on ratios is why that system has been working, so far. It's shameless how overlooked ratios are in this article. It's all about ratios, and it's not just Verizon that says so. If Cogent is out of ratio, that means they should not get settlement free peering. No matter how popular netflix is, netflix customers need to pay netflix so that netflix's ISP can pay to move the traffic. If they're sending so much traffic that it has taken ratio-based settlement free peering off the table, that just goes to show how much it costs to move that much traffic.

You know, this idea of just having everyone eat the cost of sending Netflix's data is great when it's netflix, because I'm a netflix customer and it's awesome, but when we let everyone go out of ratio on their settlement-free peers because they're ESPN's isp, or the christian streaming media blah blah blah, or whatever else you can imagine getting big and not wanting to participate in, it's going to suck. That's just turning in to cable TV all over again, with no a la carte pricing.

We pay netflix and netflix pays Cogent, sure; we pay our own ISPs so we can download (or upload) stuff, sure, but we're not paying our own ISPS to make sure that every content provider on the internet has enough bandwidth to send to everyone else. Basing settlement free peering on ratios is why that system has been working, so far. It's shameless how overlooked ratios are in this article. It's all about ratios, and it's not just Verizon that says so. If Cogent is out of ratio, that means they should not get settlement free peering. No matter how popular netflix is, netflix customers need to pay netflix so that netflix's ISP can pay to move the traffic. If they're sending so much traffic that it has taken ratio-based settlement free peering off the table, that just goes to show how much it costs to move that much traffic.

Look at it this way. Lets say I have TWC for my ISP and of course, Netflix has Cogent. Netflix is paying Cogent to move it's data around it's network. I'm paying TWC to move my data around their network.

So I request a video from Netflix. Netflix begins uploading it and Cogent moves it to TWC. Once the data is handed off to TWC, the data is basically mine. I'm paying TWC to bring it from the hand off with Cogent to my home.

Sure Cogent is handing more data to TWC than TWC is to Cogent, but both are getting paid by customers to move said data because both of them are providing a service to someone. TWC isn't supporting Cogent by receiving their data, TWC is supporting their own customers.

In the end it shouldn't matter who TWC is getting the data from, they're getting paid to take it by their own customers.

I have the same issues - I thought it's because I am overseas. Sometimes it's good, sometimes not.

For the USA I have to say the solution would appear to be simple: The ISPs need to actually deliver the speeds they have promised to their customers. As a customer I could care less what goes on behind the scenes - that's the ISPs business. All I care is that I get the 30Mbit I paid for, or whatever it is, and no matter what I do with my line. And yeah even if I decide to actually *use* 30Mbit 24/7.

More transparency in the contracts and enforcing those contracts seems all that would be required to end the shenanigans?

Basing settlement free peering on ratios is why that system has been working, so far. It's shameless how overlooked ratios are in this article.

Ratios are totally retarded when one partner only sells asymmetric connections. It is impossible to have equal traffic flow. That concept dates back to when I actually paid attention to this stuff in the last century. Know what was different then? The big ISPs then actually had both eyeball and content customers. Now we have a segregation where ISPs like Level3, Hurricane, and Cogent are primarily selling to content producers (and I use that term to cover everything from netflix, to static webpages, to xhamster, to mailchimp) and "ISPs" like TWC, VIS (VZ internet services), and AT&T are trending towards all eyeballs.

Unless they just want to exchange the same pile of money back and forth, settlement-free peering makes sense. All this dick-waving about whether the eyeballs or the content are more important is just idiotic - each side needs the other. Everyone needs to shut up, buy the equipment needed on either end, split the cost of the cross-connects, and follow best technical practices.

But I think that all the bellheads still think there's a war and they'll win it when they make their own AOL walled garden of shit.

I have no sympathy for the last mile providers. The telcos stole billions of dollars in tax breaks for things they never delivered. They barely maintain their copper plant anymore and are itching to go wireless only. The cable companies let their outside plant rot until they figured out they could sell internet for a profit so it's no surprise they're now spending billions (spread across their entire service area mind you) to bring things up to spec. They'll also whine when they start figuring out they should have pushed the fiber out further and started planning for something beyond DOCSIS 5 years ago.

These are very profitable companies in a monopoly position trying to scam their customers. They can all eat a bag of dicks as far as I'm concerned. They've all fought progress at every step of the way because they can't figure out how to make the internet metered like a phone call (telcos) or billed as a package like cable TV (yeah, cable companies).

Using Verizon's logic, they should be paying me each month. They're sending MUCH more data to my [home] network each month than I am sending to them.

Last mile network. Peering rules don't apply when you are required by the government to provide and maintain connectivity to all potential customers in an area (even if they don't use it). In these areas, the fixed costs of maintaining the line outweigh any marginal revenue gained from operating the line in the absence of subscriber fees.

A couple of years ago two of the largest ISP's in South Africa stopped peering with each other. MWeb (Independent ISP owned by the Naspers group) began to offer the first uncapped DSL package in SA, and Telkom (51% Government controlled Parastatal ISP) decided that MWebs users were going to pull too much traffic.

So they terminated their peering agreements. When a Telkom subscriber requested anything from MWebs servers (just down the road perhaps), the traffic went out through our SAT3 cable on the west coast, up to London, and then back down the Seacom cable on the east coast, into our local exchanges, and then on to the DC. And then back again. Thats a round trip of about 38,000 Km (just shy of the circumference of the Earth).

All it takes is anti-trust laws. If content services were not allowed to be ISPs, and ISPs were not allowed to be last-mile providers, the free market would straighten this whole mess out - capitalism works, if properly regulated.

The problem is that regulation to the Rethuglicans is like Holy Water to a vampire...... deadly and makes them hiss. We need to start voting out the anti-regulation idiots in office and start electing in pro-regulation (to an extent) people.

So I'm a bit late to the party here. But read my post history, I've posted extensively about this issue before and I've dealt with it professionally pretty extensively as well.

From the telco point of view (which I'm not passing judgement on here) there *are* two networks:

1. There is the national network, which the telco operates and connects every city that it has operations in. This network is run over fiber-optic lines and has interconnects out the wazoo. This network is pretty cheap to run, but needs to be incredibly reliable. Six-sigma reliable, which is kind of expensive. So they feel justified in charging net traffic providers to access this network (but more on that later.) If you look at it, the amount of money the telcos make from this kind of activity is insignificant. They basically use it to fund network upgrades.

2. There is the "last mile" network. This is what connects your house to their datacenter. This is *incredibly*, *incredibly* expensive to upgrade and maintain. Just looking at Comcast (the largest ISP in the US) their capital expenditure comes out to around $4.9 billion; most of which is spent on network upgrades. Every year. Forever. This is what your monthly cable bill goes to (well, that and content licensing fees; many of which are subsidized by HSD subscribers; but that's another matter...)

None of this is inherently bad to the consumer. However, the most damaging aspect of this arrangement comes in when you realize that, according to convention, carriers *get paid* when accepting traffic from a network. This means that, in order to maintain their balance of power and stay on the side of companies who get paid to accept traffic, most consumer ISPs have an active incentive to ensure connections remain asynchronous and weighted toward downstream bandwidth. Services that use an inordinate amount of upstream bandwidth like Dropbox, Bittorrent and Tor are discouraged and traffic shaped accordingly.

It's my job to handle this kind of crap, and you know what? I hate it. I hate it because it sucks for consumers, but at the same time, it can't be any other way. Not because these companies involved are evil, but because when you think about it as an end-to-end service, someone has to pay to upgrade the network. There are three options as to who gets to pay to upgrade the network to handle all this additional traffic from these services that consumers are using; tell me which one makes the most sense:

1. You, the consumer. Well, I mean, of course you're going to pay more, but that'll only really cover the last-mile upgrades.2. The cable/telco company. I mean, sure. This sounds fair. But it takes effort and most importantly, *money* to keep improving and maintaining the last-mile network.3. The company providing the video that you're watching. That you are either paying for or watching ads that they get paid to deliver to you.

In traditional media setups, 2 and 3 above were the same, so yeah, it's fair that the cable company pays more. But anymore, 2 and 3 are different companies. Netflix would have you believe that the choice is between 1 and 2. The cable companies want you to believe the choice is between 2 and 3. But really, the choice is between 1 and 3, because you're going to pay what you're going to pay to get to the internet, and Netflix could afford their interconnect fees if they took less profit and got fucked like everyone else paying for content rights has over the last 20 years.

Someone misses that we have given tax breaks to these big name phone and internet companies to pay for expansions to their networks over the past 30 years.So there is absolutely no reason why they should not have improved their networks. Instead, they put the money that was supposed to go to improving the networks into stockholder dividends, which should have had a bunch of people arrested and imprisoned for 10 years, at least.

So I'm a bit late to the party here. But read my post history, I've posted extensively about this issue before and I've dealt with it professionally pretty extensively as well.

From the telco point of view (which I'm not passing judgement on here) there *are* two networks:

1. There is the national network, which the telco operates and connects every city that it has operations in. This network is run over fiber-optic lines and has interconnects out the wazoo. This network is pretty cheap to run, but needs to be incredibly reliable. Six-sigma reliable, which is kind of expensive. So they feel justified in charging net traffic providers to access this network (but more on that later.) If you look at it, the amount of money the telcos make from this kind of activity is insignificant. They basically use it to fund network upgrades.

2. There is the "last mile" network. This is what connects your house to their datacenter. This is *incredibly*, *incredibly* expensive to upgrade and maintain. Just looking at Comcast (the largest ISP in the US) their capital expenditure comes out to around $4.9 billion; most of which is spent on network upgrades. Every year. Forever. This is what your monthly cable bill goes to (well, that and content licensing fees; many of which are subsidized by HSD subscribers; but that's another matter...)

None of this is inherently bad to the consumer. However, the most damaging aspect of this arrangement comes in when you realize that, according to convention, carriers *get paid* when accepting traffic from a network. This means that, in order to maintain their balance of power and stay on the side of companies who get paid to accept traffic, most consumer ISPs have an active incentive to ensure connections remain asynchronous and weighted toward downstream bandwidth. Services that use an inordinate amount of upstream bandwidth like Dropbox, Bittorrent and Tor are discouraged and traffic shaped accordingly.

It's my job to handle this kind of crap, and you know what? I hate it. I hate it because it sucks for consumers, but at the same time, it can't be any other way. Not because these companies involved are evil, but because when you think about it as an end-to-end service, someone has to pay to upgrade the network. There are three options as to who gets to pay to upgrade the network to handle all this additional traffic from these services that consumers are using; tell me which one makes the most sense:

1. You, the consumer. Well, I mean, of course you're going to pay more, but that'll only really cover the last-mile upgrades.2. The cable/telco company. I mean, sure. This sounds fair. But it takes effort and most importantly, *money* to keep improving and maintaining the last-mile network.3. The company providing the video that you're watching. That you are either paying for or watching ads that they get paid to deliver to you.

In traditional media setups, 2 and 3 above were the same, so yeah, it's fair that the cable company pays more. But anymore, 2 and 3 are different companies. Netflix would have you believe that the choice is between 1 and 2. The cable companies want you to believe the choice is between 2 and 3. But really, the choice is between 1 and 3, because you're going to pay what you're going to pay to get to the internet, and Netflix could afford their interconnect fees if they took less profit and got fucked like everyone else paying for content rights has over the last 20 years.

Someone misses that we have given tax breaks to these big name phone and internet companies to pay for expansions to their networks over the past 30 years.So there is absolutely no reason why they should not have improved their networks. Instead, they put the money that was supposed to go to improving the networks into stockholder dividends, which should have had a bunch of people arrested and imprisoned for 10 years, at least.

You seem to be under the impression that they haven't been upgrading their networks, and it's simply not true. The major telcos spend billions on network upgrades every year. Just as an example, Time Warner Cable spent $3 billion on network upgrades in 2012. They spent about the same the 2 years prior to that as well. Profits? They took $2 billion in profits last year and far less the two previous years. So how is spending more on network upgrades than they're taking in profit bad? Companies are allowed to make a profit; and none of the major telcos are sitting on exhorbitant net margins.

They are absolutely investing in upgrading their networks. It's just a long, expensive process.

Every single time using the default routes to download (view) a youtube video is slow, I use a proxy and I get fast downloads even on the highest quality. From the article and the refered reddit thread, I believe the problems lies with inefficient routing by the ISPs and the content providers.

Does anyone else think calling "slow video downloading" "suffering" is a bit of a slap in the face to half the population of this planet? Epitome of a first world problem.

This is only because the rest of the world, including (from personal experience) East Africa has equivalent or better internet accessibility and speed than North America.

So yes, it's a first world problem. In that it's a problem fairly unique to North America. It's also a problem that is relatively easily solved, but nobody elected to office has the gumption to take on the telecoms without a groundswell behind them.

Let me put it this way: in Washington D.C. there is a free, open, competitive market for internet services. I have RCN, which is cheap, reliable, and has excellent service when the reliable part falters. If you check the network identity, I'm actually on "Comcast." RCN does virtually no traffic shaping, but Comcast has recently taken to throttling the traffic to RCN customers. I noticed it because, hey my Youtube videos started failing(!) on my 75mbps connection. Started inspecting everything and discovered that Comcast was just throttling everything from Youtube onto their network. Called RCN and they said they'd speak with the Comcast techs. Within the hour everything was back to normal, although I suspect, based on long experience with Comcast, that they will start throttling again this weekend.

They do this to my parents all the time. I actually call Comcast on my way up to visit my parents and tell them to stop dicking with the bandwidth, because there are three HDTV boxes and three computers in that home, my parents pay $80 a month for internet alone and over $250 a month for Comcast's services. There shouldn't be significant lag on the internet connection when two people are watching TV (yes, true HD is pretty intense from a bandwidth perspective, but Comcast does 50% compression). The coax running from the house to the road is only 7 years old. What's worse is if I go into the service screen on one of the cable boxes I can actually see what the maximum bandwidth throughput is set to, so I know when they've actually corrected the problem. I've sat on the phone with a CS rep who just lied and lied and lied about this and finally I said "Well, I'm about done. I'm a lawyer and this call has been recorded. I'd like to speak to your division manager and I'd like a copy of your copy of this call for my records." Bandwidth doubled not even two minutes later.

I realize that companies assume stupid things. If 95% dead line assumption isn't working for them (and I don't see how it can in this day and age), then perhaps they need to reset their sights on 90% dead line? Probably 85% to be safe.

So I'm a bit late to the party here. But read my post history, I've posted extensively about this issue before and I've dealt with it professionally pretty extensively as well.

From the telco point of view (which I'm not passing judgement on here) there *are* two networks:

1. There is the national network, which the telco operates and connects every city that it has operations in. This network is run over fiber-optic lines and has interconnects out the wazoo. This network is pretty cheap to run, but needs to be incredibly reliable. Six-sigma reliable, which is kind of expensive. So they feel justified in charging net traffic providers to access this network (but more on that later.) If you look at it, the amount of money the telcos make from this kind of activity is insignificant. They basically use it to fund network upgrades.

2. There is the "last mile" network. This is what connects your house to their datacenter. This is *incredibly*, *incredibly* expensive to upgrade and maintain. Just looking at Comcast (the largest ISP in the US) their capital expenditure comes out to around $4.9 billion; most of which is spent on network upgrades. Every year. Forever. This is what your monthly cable bill goes to (well, that and content licensing fees; many of which are subsidized by HSD subscribers; but that's another matter...)

None of this is inherently bad to the consumer. However, the most damaging aspect of this arrangement comes in when you realize that, according to convention, carriers *get paid* when accepting traffic from a network. This means that, in order to maintain their balance of power and stay on the side of companies who get paid to accept traffic, most consumer ISPs have an active incentive to ensure connections remain asynchronous and weighted toward downstream bandwidth. Services that use an inordinate amount of upstream bandwidth like Dropbox, Bittorrent and Tor are discouraged and traffic shaped accordingly.

It's my job to handle this kind of crap, and you know what? I hate it. I hate it because it sucks for consumers, but at the same time, it can't be any other way. Not because these companies involved are evil, but because when you think about it as an end-to-end service, someone has to pay to upgrade the network. There are three options as to who gets to pay to upgrade the network to handle all this additional traffic from these services that consumers are using; tell me which one makes the most sense:

1. You, the consumer. Well, I mean, of course you're going to pay more, but that'll only really cover the last-mile upgrades.2. The cable/telco company. I mean, sure. This sounds fair. But it takes effort and most importantly, *money* to keep improving and maintaining the last-mile network.3. The company providing the video that you're watching. That you are either paying for or watching ads that they get paid to deliver to you.

In traditional media setups, 2 and 3 above were the same, so yeah, it's fair that the cable company pays more. But anymore, 2 and 3 are different companies. Netflix would have you believe that the choice is between 1 and 2. The cable companies want you to believe the choice is between 2 and 3. But really, the choice is between 1 and 3, because you're going to pay what you're going to pay to get to the internet, and Netflix could afford their interconnect fees if they took less profit and got fucked like everyone else paying for content rights has over the last 20 years.

Based on my readings and blogs of Level 3 comm, the largest Tier 1 in the world, they do not charge any differently between upload or download, they charge based on 95 percentile of bandwidth used.

Google had a PDF in the past year or two that was talking about peering and had a list of the top 10 networks in the world, Level 3 ranked as #1 with 15% of the Internet, #2 Global Crossing(owned by Level 3) was 5%, and #3 was Google with 3%. You can see where this is going. L3 is HUGE.

Level 3 has a lot of fun blogs and general postings around about their net neutral stance and how bandwidth is NOT an issue and their data and reasoning to support those stances.

Luckily my ISP uses L3 and also has no data caps and symmetrical bandwidth. My ISP's stances align with what L3 is saying. Why? Because L3 provides a great service and doesn't make stupid charges to my ISP for customers uploading vs downloading.

My ISP is cheaper and faster than incumbents, I would think that incumbents would have the benefit of hundreds of billions in free tax-payer grants over the decades and huge national networks that help cut out the middleman. But nope. It is cheaper for my local ISP to purchase from L3 than it is for incumbents to supply their own bandwidth, based on how much they charge for bandwidth.

Incumbents must have horrible mismanagement to charge more for less while providing worse service.

" Just looking at Comcast (the largest ISP in the US) their capital expenditure comes out to around $4.9 billion;"

That isn't much. They spend about $35b per year on operating expenses. According to case studies, going to an all fiber network reduces operating expenses about 20%. Comcast could be saving up to $7b/year, just by going to all fiber. They could SAVE almost 40% more than what they spend on upgrades, if they just upgraded to fiber.

Hence why packets are not in demand, *packet sinks* are in demand. If you think about it that way, the market makes more sense.

Excellent way to put it. The way I see it, it's my responsibility to deliver those packets as close to the "packet sinks" as possible. If I don't I'm going to get throttled at best, at worst I'm going to have an inconsistent behavior. I'm not surprised that when you occupy more than half a networks load, that someone on the last mile would say the buck stops here pal. Paying per network you transit shouldn't come as a surprise, after all there's no such thing as a centralized "internet".

Hence why packets are not in demand, *packet sinks* are in demand. If you think about it that way, the market makes more sense.

Excellent way to put it. The way I see it, it's my responsibility to deliver those packets as close to the "packet sinks" as possible. If I don't I'm going to get throttled at best, at worst I'm going to have an inconsistent behavior. I'm not surprised that when you occupy more than half a networks load, that someone on the last mile would say the buck stops here pal. Paying per network you transit shouldn't come as a surprise, after all there's no such thing as a centralized "internet".

No one "pushes" data, that's illegal, they only respond to requests. Requests drive demand, and demand pays for it.

ISPs provide a service and customers pay for it, Netflix provides a service and customers pay for it. All costs get passed along to the customer.

If ISPs want to charge customer's more for certain types of traffic, they should directly charge the customer, none of this indirectly charging via extortion bull.

People are willing to pay fair prices and the biggest ISPs are already making upwards of 90% margins. It's money printing at its best.