How Google shelters profits from content owners while building a delivery juggernaut

Synopsis

YouTube as a millstone around Google’s profitability is a mirage. Contrary to Credit Suisse’s estimates of a $470M annual loss, Google is more likely losing a fraction of that amount, due to peering for 73% of its traffic, buying bandwidth from some of the lowest-cost Tier 1 providers, using unprecedented bulk purchasing power to secure very favorable wholesale rates, and running data centers far away from

expensive locales. RampRate estimates that, based on our experience working with other top Internet, e-Commerce, and media firms, Google’s maximum loss is no more than $174M without challenging Credit Suisse’s revenue assessment. Far from being an infrastructure money pit, YouTube is key to reducing operational costs for other Google initiatives while also allowing Google to catch up to the superior network performance of competitors like Microsoft, which currently boasts 10 times as many peers and 17% fewer hops to remote reaches of the Internet.

Google is no doubt thrilled to let YouTube be known as a financial folly. In the dangerous waters of online content, a whiff of potential profit is an irresistible lure for predators such as copyright lawyers circling user generated content monetization and content partners that are all too ready to turn on their distributors in a feeding frenzy.

YouTube is a turnkey solution to a highly profitable content monetization strategy that can be unleashed as soon as Google gains stronger control of its content partners. On this point, we agree with Credit

Suisse: the unit economics work. We’ve researched and launched video monetization strategies that are profitable at a much smaller scale. Forgoing investment in the online video market because of Google’s

shell game with content providers is a decision that many prospective investors will stand to regret.

The Real Cost of Operating YouTube

In April 2009, the market saw a slew of “YouTube is doomed” or “online video is doomed” blog posts in response to Credit Suisse’s report on Google’s projected videorelated losses of $470M1. The sentiment seems reasonable at first glance – Google has been more cautious in monetizing YouTube than most observers anticipated (including RampRate). Credit Suisse’s cost analysis is thorough and close to what most large companies pay, based on RampRate’s sourcing advisory work with large, multinational infrastructure users.

However, Google is not “most large companies.” It owns a large quantity of dark fiber2, places data centers in out-of-the way geographies like Iowa and Finland3, uses commodity server equipment where

possible4, and, most importantly, uses peering agreements instead of paying for bandwidth for upwards of 73% of its traffic.5 All of these factors combine to create a likely cost basis of $414.9M, a far cry from Credit Suisse’s estimate of $711.3M. As such, using Credit Suisse’s revenue projections, we estimate that YouTube’s projected 2009 losses should be assessed at a more modest $174.2M.

The big driver in the lower estimate is in the $360.4M bandwidth cost. As Brough Turner pointed out6 soon after the initial report broke, no bandwidth cost analysis for Google is credible without including peering7. While using a peering approach has its own costs, such as connecting to a peering hub, managing complex and often contentious peer relationships, dealing with SLA-less best efforts service, and even occasional payment to powerful peers, the bulk of these costs are fixed – that is, you hire the same team and run the same fiber whether you are pushing 1 Gbps or 100 Gbps. According to Renesys, nearly 3/4 of Google’s traffic is delivered via peers – and it is probably higher considering that YouTube is known to have some peers such as AT&T8 that Renesys does not list for its corporate parent.

The top two providers of bandwidth for which Google does appear to pay transit fees are notorious for leading the charge in the price wars9 that helped push IP transit prices down by at least 20% annually even as other infrastructure markets such as co-location stabilized. When dealing with low-value content, some RampRate customers have found paths to sub-$1 / mbps bandwidth through oversubscribed connections, best-efforts SLAs and other practices straddling the borderline between paid peering and low-cost transit. Combined with the sheer amount of business that Google brings to the table, these nontraditional approaches can, if leveraged properly, reduce bandwidth costs to the $0.50 / Mbps mark.

Credit Suisse’s estimates for storage hardware also suffer from insufficient creativity — in fact, if its figures were correct, Amazon Web Services would provide storage for 50% less than Google.10 Most YouTube content is by definition “long tail,” which means the storage it requires can be of a consumergrade commodity quality. Google’s core competency is in managing commodity server / storage farms. At current rates, its servers likely cost no more than $500 apiece,11 and can add a sub-$100 1TB hard drive to produce a net cost of $.60 / GB. Even with an aggressive 1-year refresh cycle, that is just 30% of the $2.35 / GB projected by Credit Suisse. We also estimate the hosting costs are lower than projected by benchmarking an estimate of 500W of peak power use per server hosted in out of the way locations.

Upside of a $174M Loss

Regardless of how the infrastructure cost numbers are sliced, if Credit Suisse’s revenue estimates are correct,12 YouTube will take a loss in 2009 – and a significant one by most standards. So why is Google persisting with the service? There are many answers, ranging from the strategic to the inscrutable. However, one surprising factor is fundamental, no matter the revenue potential: YouTube lowers the cost of infrastructure for all other Google properties.

As discussed in the Telco 2.0 blog six months before the Credit Suisse report was published, YouTube is essential to giving Google significant leverage in peering negotiations.13 Consumer ISPs are paying backbone Tier 1 providers to carrying traffic to them. ISPs that peer directly with a major content provider of much of that traffic avoid those costs entirely. Although one might think that Google is already succeeding in that department with its other properties, the reality is that it peers with less than 1/10th the number of ISPs that Microsoft does, and takes 0.4 more hops to get to an average user.14 In other words, Google still has room to improve. YouTube is central to enabling its ability to catch up with its top rival.

Coincidentally, these providers are also the ones Google is approaching for edge caching,15 or colocation of some heavily trafficked content on servers within the ISP’s facilities. While it is not quite the sinister anti-net neutrality plot16 that The Wall Street Journal claims,17 edge caching is a sound idea that can provide a competitive advantage for many Google properties. Since most broadband providers do not provide co-location as a matter of course, the presence of the YouTube juggernaut is essential to ensuring that Google can leverage these partnerships in the same way that CDNs such as Akamai do. Edge caching is just one facet of Google’s build-focused strategy. Dark fiber costs are roughly the same whether you push 100 Mbps or 40 Gbps over each strand. Capital intensive expenditure projects such as distant data center build-outs require significant critical mass to make them worthwhile. The marginal cost tied to a property like YouTube can be negligible compared to the fixed costs that would be incurred

regardless of its presence in Google’s portfolio.

Silence Is Golden

If the YouTube strategy can be seen in this much more favorable light, why is Google not responding to Credit Suisse? The key is its lack of leverage with premium content partners, and the thousands of copyright holders whose content is posted by users onto YouTube without the owner’s permission. Any appearance of profits leads to more draconian revenue share demands from partners and additional lawsuits from owners of unlicensed content. An apparent loss deters this behavior, making it eminently advisable for Google to let the rumors of YouTube’s losses grow and compound. This perception of a loss-making business is one of the factors that contributed to ASCAP collecting only $1.6M instead of $12M from YouTube in a recent court judgment.18

The trail for this strategy was blazed long before YouTube. Apple’s poor-mouthing of iTunes served it exceptionally well for years in holding back the tide of higher revenue share demands (even as labels privately suspected the service was much more profitable than reported). The apparent stability and maturity of the business finally culminated in recent price increases.19 Google can only hope that its run with YouTube lasts as long as Apple’s luxury of $.99 pricing.

The Dark Horse Business Model

The final unspoken potential in Google’s video portfolio is the possibility of YouTube becoming a cash machine akin to Google’s search business. At 100M unique viewers per month in the US alone,20 the volume of YouTube usage is equivalent to Super Bowl-scale audiences – 365 days a year. The same force that gave TV broadcasting enough wealth to build such edifices as 30 Rockefeller Center continues to be the holy grail of advertisers today: Reach. In addition, YouTube layers on a more desired dimension to video reach: Measurability. And finally, YouTube adds two totally new dimensions: Community and

Collaboration without borders.

Google can, when it chooses, instantly ripple advertisements and direct click-to-buy opportunities across any or all of the 1.2 billion videos viewed daily.21 It can create custom formatted and branded content areas (channels) for shows, artist, films, networks and anything else imaginable. It is, at this time, the most powerful broadcasting platform in history, not to mention the world’s largest social network. All with the perfect measurability so long sought by advertisers and sellers.

So why not monetize? There are many reasons: Why pay more taxes if one is already profitable? Why get dragged into rights management issues now when one could wait until the industry is so desperate to work with YouTube that they will cave on demands? Why not wait until a certain threshold of broadband penetration? There could be many reasons, but the dark horse should not be ignored.

Conclusion

Regardless of what you may hear, YouTube costs are a fraction of any other company running similar operations. Most of Google’s bandwidth is free or near-free; its hardware is cost-optimized; and its data center costs are mostly committed or sunk. The top customers of our sourcing advisory service, whose prices are on average 20% better than the average market level, cannot deliver content as cheaply as Google’s massively scaled operation. Surprisingly enough, the ones that come closest are often those that leverage the scale of others through using cloud services. But even if a fair accounting of its costs showed a loss, YouTube gives Google the ability to achieve needed improvements in lowering cost of other operations. Loud stories about YouTube’s losses can only help deter copyright lawsuits and demands from content owners. Skepticism is warranted — but be ready for surprise news of profitability in the future.

YouTube as a millstone around Google’s profitability is a mirage. Contrary to Credit Suisse’s estimates of a $470M annual loss, Google is more likely losing a fraction of that amount, due to peering for 73% of its traffic, buying bandwidth from some of the lowest-cost Tier 1 providers, using unprecedented bulk purchasing power to secure very favorable wholesale rates, and running data centers far away from expensive locales. RampRate estimates that, based on our experience working with other top Internet, e-Commerce, and media firms, Google’s maximum loss is no more than $174M without challenging Credit Suisse’s revenue assessment. Far from being an infrastructure money pit, YouTube is key to reducing operational costs for other Google initiatives while also allowing Google to catch up to the superior network performance of competitors like Microsoft, which currently boasts 10 times as many peers and 17% fewer hops to remote reaches of the Internet.

Google is no doubt thrilled to let YouTube be known as a financial folly. In the dangerous waters of online content, a whiff of potential profit is an irresistible lure for predators such as copyright lawyers circling user generated content monetization and content partners that are all too ready to turn on their distributors in a feeding frenzy.

YouTube is a turnkey solution to a highly profitable content monetization strategy that can be unleashed as soon as Google gains stronger control of its content partners. On this point, we agree with Credit Suisse: the unit economics work. We’ve researched and launched video monetization strategies that are profitable at a much smaller scale. Forgoing investment in the online video market because of Google’s shell game with content providers is a decision that many prospective investors will stand to regret.

The Real Cost of Operating YouTube

In April 2009, the market saw a slew of “YouTube is doomed” or “online video is doomed” blog posts in response to Credit Suisse’s report on Google’s projected videorelated losses of $470M [1]. The sentiment seems reasonable at first glance – Google has been more cautious in monetizing YouTube than most observers anticipated (including RampRate). Credit Suisse’s cost analysis is thorough and close to what most large companies pay, based on RampRate’s sourcing advisory work with large, multinational infrastructure users.

However, Google is not “most large companies.” It owns a large quantity of dark fiber [2], places data centers in out-of-the way geographies like Iowa and Finland [3], uses commodity server equipment where possible [4], and, most importantly, uses peering agreements instead of paying for bandwidth for upwards of 73% of its traffic [5]. All of these factors combine to create a likely cost basis of $414.9M, a far cry from Credit Suisse’s estimate of $711.3M. As such, using Credit Suisse’s revenue projections, we estimate that YouTube’s projected 2009 losses should be assessed at a more modest $174.2M.

The big driver in the lower estimate is in the $360.4M bandwidth cost. As Brough Turner pointed out [6] soon after the initial report broke, no bandwidth cost analysis for Google is credible without including peering [7]. While using a peering approach has its own costs, such as connecting to a peering hub, managing complex and often contentious peer relationships, dealing with SLA-less best efforts service, and even occasional payment to powerful peers, the bulk of these costs are fixed – that is, you hire the same team and run the same fiber whether you are pushing 1 Gbps or 100 Gbps. According to Renesys, nearly 3/4 of Google’s traffic is delivered via peers – and it is probably higher considering that YouTube is known to have some peers such as AT&T [8] that Renesys does not list for its corporate parent.

The top two providers of bandwidth for which Google does appear to pay transit fees are notorious for leading the charge in the price wars [9] that helped push IP transit prices down by at least 20% annually even as other infrastructure markets such as co-location stabilized. When dealing with low-value content, some RampRate customers have found paths to sub-$1 / mbps bandwidth through oversubscribed connections, best-efforts SLAs and other practices straddling the borderline between paid peering and low-cost transit. Combined with the sheer amount of business that Google brings to the table, these nontraditional approaches can, if leveraged properly, reduce bandwidth costs to the $0.50 / Mbps mark.

Credit Suisse’s estimates for storage hardware also suffer from insufficient creativity — in fact, if its figures were correct, Amazon Web Services would provide storage for 50% less than Google [10]. Most YouTube content is by definition “long tail,” which means the storage it requires can be of a consumergrade commodity quality. Google’s core competency is in managing commodity server / storage farms. At current rates, its servers likely cost no more than $500 apiece [11], and can add a sub-$100 1TB hard drive to produce a net cost of $.60 / GB. Even with an aggressive 1-year refresh cycle, that is just 30% of the $2.35 / GB projected by Credit Suisse. We also estimate the hosting costs are lower than projected by benchmarking an estimate of 500W of peak power use per server hosted in out of the way locations.

Upside of a $174M Loss

Regardless of how the infrastructure cost numbers are sliced, if Credit Suisse’s revenue estimates are correct [12], YouTube will take a loss in 2009 – and a significant one by most standards. So why is Google persisting with the service? There are many answers, ranging from the strategic to the inscrutable. However, one surprising factor is fundamental, no matter the revenue potential: YouTube lowers the cost of infrastructure for all other Google properties.

As discussed in the Telco 2.0 blog six months before the Credit Suisse report was published, YouTube is essential to giving Google significant leverage in peering negotiations [13], Consumer ISPs are paying backbone Tier 1 providers to carrying traffic to them. ISPs that peer directly with a major content provider of much of that traffic avoid those costs entirely. Although one might think that Google is already succeeding in that department with its other properties, the reality is that it peers with less than 1/10th the number of ISPs that Microsoft does, and takes 0.4 more hops to get to an average user [14]. In other words, Google still has room to improve. YouTube is central to enabling its ability to catch up with its top rival.

Coincidentally, these providers are also the ones Google is approaching for edge caching [15], or colocation of some heavily trafficked content on servers within the ISP’s facilities. While it is not quite the sinister anti-net neutrality plot [16] that The Wall Street Journal claims [17], edge caching is a sound idea that can provide a competitive advantage for many Google properties. Since most broadband providers do not provide co-location as a matter of course, the presence of the YouTube juggernaut is essential to ensuring that Google can leverage these partnerships in the same way that CDNs such as Akamai do. Edge caching is just one facet of Google’s build-focused strategy. Dark fiber costs are roughly the same whether you push 100 Mbps or 40 Gbps over each strand. Capital intensive expenditure projects such as distant data center build-outs require significant critical mass to make them worthwhile. The marginal cost tied to a property like YouTube can be negligible compared to the fixed costs that would be incurred regardless of its presence in Google’s portfolio.

Silence Is Golden

If the YouTube strategy can be seen in this much more favorable light, why is Google not responding to Credit Suisse? The key is its lack of leverage with premium content partners, and the thousands of copyright holders whose content is posted by users onto YouTube without the owner’s permission. Any appearance of profits leads to more draconian revenue share demands from partners and additional lawsuits from owners of unlicensed content. An apparent loss deters this behavior, making it eminently advisable for Google to let the rumors of YouTube’s losses grow and compound. This perception of a loss-making business is one of the factors that contributed to ASCAP collecting only $1.6M instead of $12M from YouTube in a recent court judgment [18].

The trail for this strategy was blazed long before YouTube. Apple’s poor-mouthing of iTunes served it exceptionally well for years in holding back the tide of higher revenue share demands (even as labels privately suspected the service was much more profitable than reported). The apparent stability and maturity of the business finally culminated in recent price increases [19]. Google can only hope that its run with YouTube lasts as long as Apple’s luxury of $.99 pricing.

The Dark Horse Business Model

The final unspoken potential in Google’s video portfolio is the possibility of YouTube becoming a cash machine akin to Google’s search business. At 100M unique viewers per month in the US alone [20], the volume of YouTube usage is equivalent to Super Bowl-scale audiences – 365 days a year. The same force that gave TV broadcasting enough wealth to build such edifices as 30 Rockefeller Center continues to be the holy grail of advertisers today: Reach. In addition, YouTube layers on a more desired dimension to video reach: Measurability. And finally, YouTube adds two totally new dimensions: Community and Collaboration without borders.

Google can, when it chooses, instantly ripple advertisements and direct click-to-buy opportunities across any or all of the 1.2 billion videos viewed daily [21]. It can create custom formatted and branded content areas (channels) for shows, artist, films, networks and anything else imaginable. It is, at this time, the most powerful broadcasting platform in history, not to mention the world’s largest social network. All with the perfect measurability so long sought by advertisers and sellers.

So why not monetize? There are many reasons: Why pay more taxes if one is already profitable? Why get dragged into rights management issues now when one could wait until the industry is so desperate to work with YouTube that they will cave on demands? Why not wait until a certain threshold of broadband penetration? There could be many reasons, but the dark horse should not be ignored.

Conclusion

Regardless of what you may hear, YouTube costs are a fraction of any other company running similar operations. Most of Google’s bandwidth is free or near-free; its hardware is cost-optimized; and its data center costs are mostly committed or sunk. The top customers of our sourcing advisory service, whose prices are on average 20% better than the average market level, cannot deliver content as cheaply as Google’s massively scaled operation. Surprisingly enough, the ones that come closest are often those that leverage the scale of others through using cloud services. But even if a fair accounting of its costs showed a loss, YouTube gives Google the ability to achieve needed improvements in lowering cost of other operations. Loud stories about YouTube’s losses can only help deter copyright lawsuits and demands from content owners. Skepticism is warranted — but be ready for surprise news of profitability in the future.

It’s a phrase I’ve heard used more and more over the last few months, and I think it nicely sums up the general trend in datacenter computing over the last several years.

Chipset manufacturers aren’t striving to stay on the bleeding edge of Moore’s Law anymore. They realize the real-life implications of a fully loaded 40 Kw rack in the datacenter. They’re keeping their product development eyes on solutions that are more in tune with what customers need – improved computing power, but not in a hot vacuum.

Hardware manufacturers and participants across the datacenter construction chain are attuned to this, too, and have made attempts at various points to introduce the cure for feverish datacenters. Data center vendors are making a splash with the first post-mainframe water cooling revival, with some even proposing more exotic solutions like CO2-based cooling. Hot aisle/cool aisle rules of thumb are giving way to comprehensive airflow analysis using computational fluid dynamics software. Small ISPs and Internet giants alike are touting the green data center approach, whether internal (solar panels and high-efficiency UPSs) or the ability to buy greener power from a hydro- or wind-powered utility.

RampRate’s data put monthly costs per rack between $225 and $6,500 with outliers in excess of $25,000, depending on location and density. Granted, reaching those upper bounds requires cramming a 42U cabinet full of multiprocessor blades, but in a world where you can get a dual core 1U server for under $800, the days where servers were the premium product and datacenter space was commodity are long gone.

So what does this mean for datacenter customers? You have a clear need for aggressive efforts to secure the most advantageous datacenter pricing possible. You know it’s not like it was five or six years ago, where you could walk in to any datacenter decimated by the dot com bust and name your price. Now, you need to carefully identify potential providers, closely examine their pricing and any add-ons that might catch you after you sign on the dotted line. You need someone who knows the market pricing and players inside and out and has the leveraging power to get you the best deal possible from this premium market. (Of course, RampRate is perfectly positioned to do that, but to say so outright would make this a sales piece and not a blog post, so I’ll withhold my comment on that.)

What do you think? Are datacenters going to remain premium products for the long term, or will a single global round of upgrades re-commoditize space? Are OEMs doing enough with innovative designs in chipset power draws to reduce the need for absurd amounts of cooling for a single rack? Will this idea of the ‘greening’ of the datacenter take off – and, if so, will that create a new subset of elite premium datacenters?