It’s not a “fast lane” but Comcast built a CDN to charge for video delivery

Comcast's last mile video delivery service brings in cash from broadcasters.

Comcast is now competing against content delivery networks (CDNs) such as Akamai with a new service that can improve delivery of video to Comcast subscribers in exchange for payment.

"This new offering allows content owners to go directly to the ISP and have their content stored and delivered via the last mile, thereby displacing some traffic currently delivered by third-party CDNs like Akamai and Limelight Networks," well-connected industry analyst Dan Rayburn reported today. "While this is the same type of CDN service that other commercial CDNs like Akamai already offer, Comcast can offer a very good SLA since they are a last mile provider."

Further Reading

When Netflix and Comcast customers both pay Comcast, the traffic flows smoothly.

Comcast didn't issue an announcement and hasn't responded to a request for comment from Ars. The Information reported that "John Schanz, Comcast’s chief network officer, said in an interview that the offering is currently being tested with some customers and is expected to roll out broadly if all goes well." The service will let Web content companies "bypass network middlemen and deliver their services directly to Comcast Internet customers," The Information article said.

The service is already live and has a few paying customers delivering content on Comcast's CDN, Rayburn reported. "Right now, Comcast’s CDN service is primarily targeting large and mid-sized content owners for large file downloads and streaming of video, both live and on demand," he wrote. "While Comcast can also deliver content for smaller content owners, it does not make sense for the majority of small content owners to go direct to an ISP for content delivery. Comcast isn’t offering what the CDN industry calls value add services, so CDNs like Akamai that get a large percentage of their revenue from services like dynamic site acceleration won’t see that portion of their business affected."

The service is being deployed at a time when the Federal Communications Commission is debating whether net neutrality rules should allow Internet "fast lanes," in which Web services pay Internet service providers for a faster path to consumers. Comcast argues that it is allowed to offer fast lanes, also known as "paid prioritization," despite making net neutrality agreements when it acquired NBCUniversal.

Despite making that argument, Comcast has said it doesn't intend to offer fast lanes. And the new CDN service is not a fast lane, Rayburn told Ars in a phone interview.

"There's no guaranteed prioritization of any kind," Rayburn said. "This is simply a content delivery offering just like an Akamai or someone else sells, but the difference here is the company offering the service owns the network."

Video providers typically pay third-party CDNs to ensure that traffic is delivered along the most optimal route to each consumer and that heavily accessed content is cached for quick retrieval. CDNs themselves often pay ISPs for direct access to their networks. When Netflix built its own CDN, it tried to get free connections to ISP networks. While Netflix succeeded in some cases, it was forced to pay both Verizon and Comcast.

While no specific Comcast CDN customers were revealed, TV broadcasters who also deliver streaming video over the Internet are likely suspects. "Their ideal customers for the new service would be broadcasters, gaming companies, software companies, large publishers, and those in the media and entertainment vertical," Rayburn wrote.

Rayburn said businesses could save 20 to 40 percent a year by paying Comcast instead of a third-party CDN. For instance, if a customer is spending $1 million a year on CDNs and sends 30 percent of its traffic to Comcast, that could end up being about a $100,000 savings per year, he said.

"Let's say that you're a broadcaster. You're not big enough to build out your own CDN, you're not a Netflix, you're not a Google, so you're going not going to build out your own CDN. You're going to use third-party commercial CDNs," he told Ars. If 30 percent of a video provider's content is going to Comcast customers, "you could take that 30 percent of your traffic, you could have Comcast deliver it for you and you could pay a much cheaper price because Comcast owns the network and it's wholesale, so they're selling it at a discounted rate compared to a CDN like Akamai or someone else who doesn't actually own the network."

"There's no question that the ISP should be able to give you a better SLA [service level agreement] because they own the network. But a better SLA doesn't guarantee you QOS [quality of service]. Those are two different things," Rayburn said. "If you're a third-party CDN you can't guarantee that the network will be up 100 percent of the time because you don't own or operate the network, whereas an ISP could guarantee you something like 100 percent network uptime because they're the ones who actually own it, they might feel confident enough in saying, 'yeah we could offer that.'"

Comcast's long game

Comcast is unique among US ISPs in that it offers a commercial CDN directly to content owners without licensing a platform from a third party, Rayburn wrote, noting that AT&T resells Akamai services. Verizon struck a deal with HBO to deliver content inside the last mile for FiOS subscribers in 2010, but its CDN plans stalled temporarily. Verizon is expected to get back in the market due to its recent acquisition of EdgeCast.

"No small content owner builds their own CDN, so small content owners don’t need to go direct to any ISP for CDN services or interconnect deals," he wrote. "Small content owners will always use third-party commercial CDNs as it’s cheap and they can get good quality delivery from a single provider. That’s why the only companies doing interconnect deals are content owners like Microsoft, Google, Netflix, Apple, Yahoo! etc. because they have built their own CDNs. The reason Comcast is doing this is to open up another avenue for any size of content owner to deliver content."

Some network neutrality proponents have asked the FCC to examine interconnection deals as part of the Comcast/Time Warner Cable merger review. Comcast's CDN service could be another item on the list of things merger observers want the FCC to examine.

"Like with all these issues, the question is whether Comcast is abusing its market power and charging other networks for access to its customers," Public Knowledge Senior Staff Attorney John Bergmayer told Ars. "If it is, it doesn't matter whether the interconnection is called a 'CDN,' 'peering,' 'transit,' or whatever. There's nothing inherently wrong with an ISP running its own CDN, but with Comcast it's all about the long game."

Promoted Comments

In theory I would have no issues with this as long as the ISP didn't treat their own CDN better than other CDNs, but this is Comcast. You know they're going to subsidize costs from their customers to strong arm competition for this "competing" service.

If this Comcast CDN service was an entirely separate company with its own accounting and got no money from the parent company while being strongly audited and made sure not to get preferential treatment from the parent company, I would have little issue.

This is what a monopoly can do. Start providing other services at a lower price than any competition.

Even if they weren't a monopoly, they could offer this service at a lower price than any competition. No one has better access to Comcast's customers than Comcast, and they don't have to pay for the bandwidth.

Being a monopoly just makes it possible to basically force people into using it.

In theory I would have no issues with this as long as the ISP didn't treat their own CDN better than other CDNs, but this is Comcast. You know they're going to subsidize costs from their customers to strong arm competition for this "competing" service.

If this Comcast CDN service was an entirely separate company with its own accounting and got no money from the parent company while being strongly audited and made sure not to get preferential treatment from the parent company, I would have little issue.

Am I understanding correctly that basically I will be charged more if I want fast video delivery?

The frustrating thing about "fast" video delivery is that it really either works or it doesn't. I just need it fast enough so it won't buffer while I watch. Does that mean for any Comcast customer who wishes to stream HD video via netflix/amazon etc an add-on package will have to be purchased?? wtf

I don't disagree, but would point out that every ISP has a monopoly on the last mile to their own customers.

That's asinine on its face - in most other countries, people can change their ISP is they're unhappy. The US is basically a continental duopoly across its states, which means that both companies can essentially collude and force the other players int he market to pay them more money for something that the customer already pays for.

In theory I would have no issues with this as long as the ISP didn't treat their own CDN better than other CDNs, but this is Comcast. You know they're going to subsidize costs from their customers to strong arm competition for this "competing" service.

If this Comcast CDN service was an entirely separate company with its own accounting and got no money from the parent company while being strongly audited and made sure not to get preferential treatment from the parent company, I would have little issue.

Well by the description, the CDN is inherently both better and worse than a general CDN accessible from the whole internet. It is better for supplying data to Comcast's customers, because once the data is in the CDN it doesn't have to cross the barrier into the Comcast network again. It is, from the end-user's perspective, going to be monumentally faster than general CDN's. However, it's also worse, because it ONLY works for Comcast customers. Even if you could get to it from outside the Comcast network, you almost certainly wouldn't want to, as we've established that Comcast has horribly insuffecient bandwidth to the rest of the internet (if they had just spent the money for the bandwidth infrastructure they need they wouldn't be trying to break net neutrality in the first place)

Well, a ISP can offer any service they want, they have the right to do so as a company and have already for years offered other services, most ISPs also offer usually domains, hosting and email account to customers, cable, etc.

So its not wrong if some ISP offers a CDN service at all.

What is wrong is if they start to put "other" CDN providers into a slower lane and give higher priority to their service. That is indeed not only unfair competition but a monopoly and this is exactly what people where discussing the FCC should never have approved. This is what net neutrality is all about. Competition is fine and welcome, but unfair competition is not if you are abusing your market position. And in most countries one or two ISP have most of the users so they are in a very abusing position already in terms of market.

Its not that ISPs cannot offer other services, they can. The problem is when they create a fast lane, this means that a slow lane for others services exist already. The FCC must be idiots because you cannot have a fast lane without a slow lane, its not possible otherwise the fast lane would not be fast. Guess which lane is going to get bottlenecks...

The FCC has basically destroyed the Internet allowing them to do so. ISP will now of course prioritize traffic for their own services and from companies like Netflix that pay them and at the same time maybe even offer a special service to customers for higher access as well splitting the Internet in half, those that pay more and those that don´t.

The problem is that this creates unfair competition for startups and other services which rely on ISP to deliver their end services to customers. The time where anyone could register a domain and start its Internet business online is over as they will not be able to be in those fast lanes...

Lets see how much time goes on until this ISP and the FCC is sued by groups of smaller companies. The FCC is supposed to protect innovation, and the US leading Internet business should put even more priority in neutrality to promote grow of more digital business. This was the worse thing the FCC has done so far. They will hurt the US economy big time when startups start to grow overseas.

AAANnndd this is why it should be regulated. If comcast wants to be in the content industry, it needs to be seperated from the internet (data transport) industry or forced to refrain from anti-competitive actions.

Comcast has ALWAYS had a CDN -- they just used Akamai's technology to implement it. If they have transitioned to their own CDN, it is probably because they got tired of paying Akamai's exorbitant costs...

The end of the internet began with that FCC decision, opening the way to crap like this.

If you want to deliver a large amount of data to 'internet' users we can no longer just go to a CDN and pay for it. We'll have to sign up for Comcast's, AT&T's, Verizon's, Charter's and then finally Akamai so that the rest works well too. Our CDN fees quadrupled boys. If you want to get around it by building your own then you have to pay for the fast lanes.

Comcast has ALWAYS had a CDN -- they just used Akamai's technology to implement it. If they have transitioned to their own CDN, it is probably because they got tired of paying Akamai's exorbitant costs...

Okay, so it's not good for Comcast to pay exorbitant costs, but it's okay for them to charge said costs? Uh, no.

I don't disagree, but would point out that every ISP has a monopoly on the last mile to their own customers.

That's asinine on its face - in most other countries, people can change their ISP is they're unhappy. The US is basically a continental duopoly across its states, which means that both companies can essentially collude and force the other players int he market to pay them more money for something that the customer already pays for.

Except that this affects content providers more than customers, and from the tone of the article, positively. If a content provider is paying 100% of their distribution to Akamai as their CDN, with 30% of that traffic going to Comcast subscribers, they could split their distribution between Akamai and Comcast CDN. Comcast would charge less than Akamai for service to their ISP customers and have the added benefit of improved performance.

This would actually be a reason for a customer to NOT leave Comcast, since the result of this would ostensibly be better video streaming performance (from the big sites, anyway).

Oh look. They are trying to get BOTH vertical AND horizontal monopoly status. Surprise surprise. So, post-merger, Comcast will be fast on its way to being the only broadband ISP nationwide or as good as with its massive market share. Furthermore, Comcast develops the content, transmits the content and with this CDN stores the content. Everything is Comcast at every level. And to think, I used to worry about America becoming Walmartia. Welcome to Comcastistan.

If you want to deliver a large amount of data to 'internet' users we can no longer just go to a CDN and pay for it. We'll have to sign up for Comcast's, AT&T's, Verizon's, Charter's and then finally Akamai so that the rest works well too. Our CDN fees quadrupled boys. If you want to get around it by building your own then you have to pay for the fast lanes.

Actually Akamai's should be sufficient. I haven't see any reports anywhere of Akamai, or any other CDN, having performance problems for Comcast's or an other ISP's customers. That's probably because Akamai pays Comcast for their peering or Colocated servers like they always have and aren't trying to get a free ride. The performance problems have been with backbone providers who tried to take on Netflix's huge amount of traffic and assume they could just get the added capacity they needed with free peering with Comcast.

This distinction is important and kind of technical. Obviously implementing "fast lanes" by definition moves everything else to the "slow lane". Nobody is disputing that. So if Comcast was running an Internet accessible, general purpose CDN and prioritizing traffic requested by their customers as it was coming into their own network, that would be a violation of net neutrality.

But hosting something like this internally is a different matter. Net Neutrality concerns data flowing from the internet to the ISP's network and back. That traffic must be treated the same, regardless of what it is or where it is coming from. But once the data is inside the network, it is no longer in the scope of net neutrality, because it's no longer on the internet (although it seems to be from the end user's point of view).

Now should ISP's be running CDN's? I'm not sure. It seems like that has the potential for monopolistic behavior. Perhaps it should be a regulation that ISP's must allow arbitrary CDN's to run endpoints on their internal networks, assuming they pass some sort of publicly available validation test to ensure they don't break the network (much like how anything can be attached to the old copper phone network if it doesn't break the network).

I'm sure this discussion will end up as another Comcast is evil debate but I don't know that I see anything wrong with this.

The wrong comes from the fact that they have a complete choke-hold on the end customer aka THE market. There is no competition. None. Zip, Zilch... ad nauseam. And as Comcast just got bigger, as will happen until we've got a tiny group of players that will control the entire Internet as far as the U.S. goes, that is essentially government guaranteed( starting from the idiotic way cable and other communications was done with exclusive deals per town, etc., now from the outright ownership these companies enjoy of the government itself ).

Communications should be nationalized, along with every other service critical to a functioning first-world society.

This distinction is important and kind of technical. Obviously implementing "fast lanes" by definition moves everything else to the "slow lane".

How does implementing a fast lane move everything else to a slow lane? When the government adds a lane to a freeway, it doesn't make the existing lanes go slower, it makes them go faster, even if the new lane charges a toll like I 880 in the East Frisco Bay.

President Bush called this "making the pie higher" but it seems hard to understand.

if you have up to say 100mbps dedicated for consumer bandwidth and you sell dedicated bandwidth on those lines, you are going to slow others down

This distinction is important and kind of technical. Obviously implementing "fast lanes" by definition moves everything else to the "slow lane".

How does implementing a fast lane move everything else to a slow lane? When the government adds a lane to a freeway, it doesn't make the existing lanes go slower, it makes them go faster, even if the new lane charges a toll like I 880 in the East Frisco Bay.

President Bush called this "making the pie higher" but it seems hard to understand.

Road metaphors don't work with this. It's more like standing in line to get into a party. In a neutral situation, it's first come first serve. If instead you let everyone with VIP status cut the line, then anyone who doesn't have VIP status will take longer to get in (how much longer will depend on how many VIP's there are).

Of course, that's only a problem if there's a line. If ISP's would actually build out the infrastructure they need to service the customers they have, then it wouldn't matter, because there wouldn't be a bottleneck, and violating net neutrality to prioritize certain types of traffic over others would accomplish nothing except useless overhead.

if you have up to say 100mbps dedicated for consumer bandwidth and you sell dedicated bandwidth on those lines, you are going to slow others down

Even in that extreme case, your logic doesn't work. Imagine two streams in the neighborhood network, one a Vonage stream and the other a Microsoft patch update. The Vonage stream wants low latency, which can be accomplished by moving packets around. As long as the arrival time of the last MS packet isn't altered by inserting some Vonage packets ahead of some MS packets, nobody has been slowed down but somebody has been sped up. And the arrival time of the last MS packet is determined by the total volume of Vonage packets, not their rearrangement in the queue.

It's not too complicated, is it?

say netflix, apple, google and who ever else buy up 80% of the max bandwidth. their data gets good performanceamazon doesn't pay up and Amazon prime videos will crawlor vudu doesn't pay up and if you want to rent a movie it's either itunes or xfinity or watch vudu buffer all the time

Road metaphors don't work with this. It's more like standing in line to get into a party. In a neutral situation, it's first come first serve. If instead you let everyone with VIP status cut the line, then anyone who doesn't have VIP status will take longer to get in (how much longer will depend on how many VIP's there are).

Before discrimination, there is one line for everyone. After discrimination, there are two lines, one for VIPs and one for the common rabble. Taking the VIPs out of the rabble line makes the rabble line move quicker.

Your basic problem is failure to understand that total capacity always increases. In the US, broadband capacity doubles every three years, and has since 1998.

and so does the amount of data sent and the same companies will buy up the new bandwidth. and they will send the data at the highest quality to use up all the bandwidth so that there is none left for the competition

say netflix, apple, google and who ever else buy up 80% of the max bandwidth. their data gets good performanceamazon doesn't pay up and Amazon prime videos will crawlor vudu doesn't pay up and if you want to rent a movie it's either itunes or xfinity or watch vudu buffer all the time

In the real world, Netflix pays for new capacity. Little companies don't do that because they're fine with the standard level of capacity and service.

and so does the amount of data sent and the same companies will buy up the new bandwidth. and they will send the data at the highest quality to use up all the bandwidth so that there is none left for the competition

Can you give me an example of when that has ever happened? When Netflix started peering with Comcast, Level 3 and Cogent's connections got faster as they were moving less data over them.

As soon as I started reading the article, I didn't get very far before I thought to myself, "This is very troubling. Top-to-bottom vertical integration of internet content delivery. Comcast wants to own the internet."

and so does the amount of data sent and the same companies will buy up the new bandwidth. and they will send the data at the highest quality to use up all the bandwidth so that there is none left for the competition

Can you give me an example of when that has ever happened? When Netflix started peering with Comcast, Level 3 and Cogent's connections got faster as they were moving less data over them.

this is completely different than peering

we have had the tech for a long time to take a data circuit and in real time not only track the data at the application level going through it, but set the bandwidth depending on the application or sender. it just hasn't been used yet at the ISP level to sell dedicated bandwidth to senders of data

but i've seen adds from other countries where an ISP will sell a basic internet package and if you pay more you get access to different sites or their performance will increase. same concept except in this case comcast might charge those companies

Road metaphors don't work with this. It's more like standing in line to get into a party. In a neutral situation, it's first come first serve. If instead you let everyone with VIP status cut the line, then anyone who doesn't have VIP status will take longer to get in (how much longer will depend on how many VIP's there are).

Before discrimination, there is one line for everyone. After discrimination, there are two lines, one for VIPs and one for the common rabble. Taking the VIPs out of the rabble line makes the rabble line move quicker.

Your basic problem is failure to understand that total capacity always increases. In the US, broadband capacity doubles every three years, and has since 1998.

My point was that CDN's running on an internal ISP network for the use of the ISP's customers's only isn't a violation of net neutrality, because it doesn't involve the border between the internet and the ISP's network.