Comcast has quietly entered the CDN market with a commercial content delivery service primarily targeting medium and large size content owners. The service is live and the company has a few paying customers already delivering their content on Comcast’s CDN. This new offering allows content owners to go directly to the ISP and have their content stored and delivered via the last mile, thereby displacing some traffic currently delivered by third-party CDNs like Akamai and Limelight Networks. While this is the same type of CDN service that other commercial CDNs like Akamai already offer, Comcast can offer a very good SLA and pricing, since they own the network. This is similar to what Level 3 did, when in 2010, they came to the market with a CDN service that was priced lower than others.

[Updated 4:06pm: Just in case it wasn’t clear to some readers, Comcast’s CDN service is not providing any kind of “fast lane” for content or any type of “prioritization” of content. A well operated third-party CDN will have similar performance to Comcast’s CDN and user experience will mostly be imperceivable by the average consumer. The service also has nothing to do with net-neutrality or interconnect agreements. Those who want to imply or suggest otherwise are doing so incorrectly.]

Because Comcast owns the last mile, and commercial CDNs don’t, content owners who go direct to Comcast will be able to get a cheaper price for CDN services. While contract variables like volume and length of contract will dictate price, I expect some content owners to be able to pay 20%-40% less than what they pay now. Since Comcast owns the network and has a lower cost, they can offer a lower price in the market. Right now, Comcast’s CDN service is primarily targeting large and mid-sized content owners for large file downloads and streaming of video, both live and on demand. While Comcast can also deliver content for smaller content owners, it does not make sense for the majority of small content owners to go direct to an ISP for content delivery. Comcast isn’t offering what the CDN industry calls value add services, so CDNs like Akamai that get a large percentage of their revenue from services like dynamic site acceleration won’t see that portion of their business affected.

Comcast’s new service is considered an on-net CDN offering as Comcast is primarily delivering content via their last mile. While they can also support off-net delivery outside of their network footprint, that’s not the business they are targeting. For content owners that have a large percentage of their content going to Comcast subscribers, it would make sense for some of them to take that portion of their traffic off a third party CDN and move it over to Comcast. While Comcast’s offering will be attractive to large content owners, this does not mean ever large content owner will use it. Many will find it as an attractive option in the market as a large portion of major content owners already use a multi-CDN approach to delivering their content. So if they can have Comcast pull content from their origin storage and deliver it locally for Comcast subscribers, and at a discounted price, many will the see the value considering that Comcast subs probably account for about 30% of their overall traffic.

As the news of Comcast’s commercial CDN offering gets out, I’m sure some are going to suggest that Comcast is doing this to counter the arguments that paid interconnect deals, like the one they have with Netflix, is bad for the little guy. By Comcast now offering content owners the ability to deliver their content directly inside Comcast’s network, this now opens up the service for smaller content owners. The problem with that argument is that it is dead wrong. No small content owner builds their own CDN, so small content owners don’t need to go direct to any ISP for CDNs services or interconnect deals. Small content owners will always use third party commercial CDNs as it’s cheap and they can get good quality delivery from a single provider. That’s why the only companies doing interconnect deals are content owners like Microsoft, Google, Netflix, Apple, Yahoo! etc. because they have built their own CDNs. The reason Comcast is doing this is to open up another commercial option for any content owner who wants to deliver content.

Comcast’s new CDN offering has absolutely nothing to do with interconnect relationships and Comcast isn’t targeting small content owners or those who have already built their own CDN. Their ideal customer for the new service would be broadcasters, gaming companies, software companies, large publishers and those in the media and entertainment vertical. What Comcast is doing with their commercial CDN offering may sound like a new idea to many, but it’s not. In 2010, Verizon did a deal with HBO to deliver their content inside Verizon’s last mile for FiOS subscribers. This was due to the fact that Verizon had built out their own CDN, using the technology from Velocix, which was acquired by Alcatel Lucent. Verizon didn’t do many deals with content owners and ended up changing their CDN plans a few years back, but I do expect Verizon to once again enter the market and offer a similar service to Comcast’s, due to Verizon’s recent purchase of EdgeCast.

Since the commercial CDN market for any ISP is small when compared to their core business, some will ask why Comcast or any ISP even wants to be in the commercial CDN business to being with. The key thing to remember about commercial CDNs is that many times they make cost trade offs around performance. Adding a disk drive to a cache to improve hit rate or a new server in a location does not have the same financial drivers as it does for any ISP. Typically a CDN pays the same $/Mbps to serve a city whether it is sourced locally or from across the country. An ISP’s network costs to serve from long distance drives financial incentives to both add local storage and server capacity to keep traffic local. Some CDNs deliver a large portion of their traffic from sub-optimal locations to save on server and storage costs.

Right now, Comcast is the only ISP in the U.S. to offer a commercial CDN service directly to content owners via their own in-house solution, as opposed to licesing a platform from a third party. While some might suggest that ISPs offering commercial CDN services will be a trend moving forward, it isn’t. The only three ISPs this kind of service would even make sense for would be Comcast, Verizon and AT&T, and AT&T has already given up on their own internal CDN and simply resells Akamai.

How does an ISP on-net CDN service differ technically from the paid prioritization arrangements that are currently at issue in the network neutrality debate? (Or does it?)

I think the answer is that in the CDN case, most of the content provider’s traffic doesn’t actually pass through the ISP’s network along with other content, just a message to the appropriate edge cache to serve it to the end-user. With paid prioritization, all the traffic goes through the network but routers are instructed to give it priority over other packets.

If this is essentially correct, my main question is why (a) the ISP and (b) the content provider would prefer one arrangement over the other, purely as business/performance matter.

Amiram

From a technical standpoint, they are different. I would call the CDN method Passive Prioritization and the other method Active Prioritization.

With the latter, ISP network devices are instructed to prioritize certain network traffic, effectively delaying other traffic. Packets arrive at a buffer on a network device, instead of being sent along FIFO, some are actively moved to the head of the queue. When congestion occurs within the ISP network the prioritized traffic will be served first and theoretically provide better service to the end user vs. other services, which will suffer.
However, everyone suffers anyway when there is congestion. When some services are prioritized the no-prioritized services suffer some more. This is mostly noticeable on video, online gaming and to some extent on file sharing services.
Another caveat is that, with paid prioritization alone as described above, the ISP doesn’t have control over the delivery outside their network. If transit and peering connection are not scaled and prioritized and ISP cannot assure end-to-end quality of service under this scheme.
Does “Active Prioritization” really exist? Or is it just code word for ISPs to get out of the way?
If one builds their own CDN or even use Internet CDNs effectively (Akamai, Limelight, etc.) one should be able to provide high quality of service to one’s subscribers, assuming they are subscribed to an ISP which scales their network to subscriber demand. Comcast (and other ISPs) have been accused of actively throttling Netflix (and P2P traffic) or refusing to scale peering on their end in order to force Netflix into an agreement to fix a problem that Comcast allegedly caused by its own action or inaction.

With ISP CDN, or “Passive Prioritization” if you’d like, content is relocated closer to the end user, most likely on lower RTT links which are less likely to have choke points since there is no peering involved and the ISP has full control of the network over which traffic is served and can potentially resolve any bottlenecks independently. Services will inherently enjoy better bandwidth and, when other off network service suffer from peering or transit issues, will provide better service to the end user.

To combine the two methods – An ISP controls both ends – Sender and Receiver – and therefore can also add prioritization to overcome congestion conditions within his own network and truly provide end-to-end QoS and essentially a “fast lane” for its CDN customer.

I am guessing the Netflix-Comcast deal is a combination of the two methods, but we can’t know for sure since it isn’t public.

I can think of a few reasons why go one way or another or both. While I believe ISPs going into the CDN business is legitimate, has not been challenged to data AFAIK and won’t be contested in court or by the FCC, prioritization on network devices may very well be contested in courts and subject to regulation – I assume most ISPs and content provides would prefer to stay away from employing Active Prioritization techniques.

http://hightechforum.org/ Richard Bennett

Well, Kevin, if you read the advertising claims for Value-Add CDNs such as the InstartLogic Global Accelerator used by the Washington Post ( http://instartlogic.com/service/global-network-accelerator/ ) and the CacheFly system used by Ars Technica ( http://www.cachefly.com/ ) either there’s no difference between these services and the “fast lane” or the VACDNs are guilty of false advertising. CacheFly promises “Up to 10X faster performance”, period. Compared to what, I don’t know.

It’s bit difficult to map the notion of the Magic Gateway that Tim Wu describes in “Network Neutrality and Broadband Discrimination” to the real world because ISP networks connect to the rest of the Internet through thousands of Ethernet ports, each of which is in various states of congestion at any given time. So traffic going over a lightly loaded port will move with less latency than traffic over a heavily loaded port of the same capacity in any given second, but the pattern can reverse in a matter of seconds, minutes, or hours. While it’s stable, the lower traffic port is effectively higher priority than the higher traffic port.

Each port feeds a middle mile path, and the paths are part of a humungous mesh that has neighborhood head ends at the opposite edge from the Internet side. So the most straightforward way to build a fast lane is to provision lightly used paths for priority traffic, but that’s not enough to meet an SLA.

So the second level is to actively re-order queues to limit dwell time for each stream, which doesn’t always mean anyone gets delayed in any meaningful sense. If the queue has a 100 file transfer packets ahead of one voice packet and 100 more after it, moving the voice packet to the head of the queue doesn’t slow down the file transfer operation. It’s not done until the last packet arrives in any case. So you can see discrimination at the packet level that has no effect on the operational level. A CDN is a blunt instrument compared to queue ordering because it deals with congestion on a longer time line.

So it really comes down to the SLA the carrier is trying to meet. CDNs are mainly for streaming apps that are relatively more resilient to latency than, say, voice or conferencing apps. If an ISP wants to let people use Cisco Telepresence in their homes, they will need to provide some non-standard plumbing between the ports in the IX and the neighborhood head end. The neighborhood network is fine in most cases, but if it’s not, DOCSIS is easy to prioritize.

Mr. sudafed

Dan you are overlooking that Comcast peering is extremely congested due to tolls they set to prevent competition. Obviously they are going into this with an unfair advantage. Don’t take my word for it, Google for Comcast and “Backdoor Santa” for details.

danrayburn

They do not have an “unfair advantage”. Yes, their cost is lower becasue they own the network, but if that is unfair, then Level 3, AT&T, Tata and all the other ISPs that offer commercial CDN services are also being “unfair”. Has nothing to do with peering. It’s a commercial CDN service.

Claritin

Did you read the Backdoor Santa NANOG postings? Thịs is some good insight. Running a CDN gives comcast the advantage in that they do not need to charge for delivery, just compute.

Kirby

Wasn’t that in 2010? I think the ISP service is mostly good and lots of other sites’ performance agrees with that. Perhaps that single graph was a “highlighted exception” to make a statement and there are probably hundreds to thousands of other graphs which look just fine.

Also which “path” the CDN chooses is their decision. If they choose to use “Backdoor Santa” path, yep.. their service will suck. If they choose to avoid “Backdoor Santa” they will give a good service to their customers.

Also if you are a large traffic generator, you can *create* “Backdoor Santa” congestion pretty easily.

terry305

As someone working at a hosting company, let me tell you, the “Backdoor Santa” graphs are the norm more than exception. It is impossible to buy bandwidth from anyone with capacity into Comcast, even after they made a deal with Netflix.

I do not expect that the Comcast CDN will be a product competitive to LimeLight or Akamai because they are congested into so many networks. For a CDN to run well, you need to have clear paths.

Real Santa

I think you loosely use the word “impossible” and find that hard to believe given performance as of late. Care to back that statement up with recent graphs?

Santa Coming to Town

When you deliberately congest peering and then offer a CDN service, you are making this about peering. I strongly suggest learning something about this topic – it is now a big deal. Come to the NANOG peering track – it will be a major item of discussion. Or keep peddling this misinformation.

http://www.ivpcapital.com/blog Michael Elling

Dan, it is not at all similar to what Level3 did in 2010. The latter’s CDN was moving up the stack from the transport to the control layers in the WAN; from layers 1-2 to layers 3-4 (CDNs are just tradeoffs between transport capacity and switching/storage influenced by time or density issues). Comcast, by your own admission of focusing on last-mile traffic (on-net opportunities) is simply moving the vertical boundaries (WAN/MAN demarc) to the core. It is extending last-mile layer 1-2 monopolies into the core (a physical bottleneck) because it cannot move up the stack due to the nature of TCP/IP. It has nothing to do with economic benefit and efficiency. Just the opposite in fact.

Please refer to my InfoStack framework on my website under models as it provides a useful way to understand and capture past, present and future business models and policy issues across service layer, geodensity, and market segment axis, while simultaneously dealing with supply convergence and demand divergence

All that said, I am actually in favor of some type of 2-way or bilateral settlements and prioritization. TCP/IP lacks price signals and incentives. It is dominated by advertising led revenues (in turn dominated by Google with access to our private data and usage), private enterprise solutions and edge-carrier interconnections. The core has scaled, but is not scaling well at the edge; with the exception of 802.11/ethernet. This inability for the internet to scale at the edge is solved by price signals which clear supply and demand north-south and east-west in the framework. A market-driven “balanced settlement” exchange model that would replace govt mandated access charges can only work if set by competitive market forces. And the latter can only work if interconnection occurs somewhere in the last mile.

Where that interconnection occurs is the issue. Some say that the cloud has already been replaced by “the fog.” We were very fortunate as an industry and economy that Steve Jobs was able to force interconnection in the mobile device on AT&T in 2007 because he could take advantage of a prior open-access or interconnection policy called part 15 that has a rich 50 year history and which morphed into 802.11; aka Wifi or layer 2 offload, which today accounts for, or affords, 70% of consumption across those devices. Had it not been for that, we certainly would not have had the smartphone boom, and explosion in the application ecosystems which fueled investment in the cloud and the industry you follow.

Cause and effect and unintended consequences are rife in the framework across horizontal layers and vertical boundaries; themselves, highly fluid and contextually defined by the supply/demand relationships and push/pull dynamics.

Daily posts by Dan Rayburn about the online video industry, business trends & analysis, market data & research as well as the online video business models in the media & entertainment, broadcast, advertising & enterprise industries.