Posted
by
samzenpus
on Thursday March 18, 2010 @10:55AM
from the self-sustaining dept.

MrShaggy writes "Credit Suisse made headlines this summer when it estimated that YouTube was costing Google a half a billion dollars in 2009 as it streamed 75 billion videos. But a new report from Arbor Networks suggests that even though Google is approaching 10 percent of the net's traffic, it's got so much fiber optic cable it is simply trading traffic, with no payment involved, with the net's largest ISPs. 'I think Google's transit costs are close to zero,' said Craig Labovitz, the chief scientist for Arbor Networks and a longtime internet researcher. Arbor Networks, which sells network monitoring equipment used by about 70 percent of the net's ISPs, likely knows more about the net's ebbs and flows than anyone outside of the National Security Agency."

Depending on the terms, it means Google can also act as a peering or transit point between these companies and or even have an IXP's at their locations, so theres incentive for ISP's to sign up beneficial transit agreement, especially considering Google has data centers around the world. Google has more power than Tier 1 ISP's alone. The article's note about "serving customers YouTube faster" is a moot point - Google's infrastructure and routing contracts alone act as a great incentive for ISP's to make a peering agreement with Google.

Unless you count the cost of running the fiber, and the cost of routers and maintenance. And the cost of generators, and power and other operating costs... Basically, "much" is relative. Compared to "buying" the bandwidth from a Tier-1 provider, probably not much. Compared to 0, probably very much...

It points out the folly when people say "Comcast/AT&T/Verizon/whomever has to pay huge upstream bandwidth costs, bandwidth isn't free y'know!", and it always gets marked as insightful.

These guys are so large, bandwidth, other than physical maintenance of their physical plant, isn't a big part of their expenses. When Comcast says "We need to limit bandwidth because of those evil hackers", that's code for "I don't feel like rolling out DOCIS 3 for a few years". When AT&T Mobile says "Those iPhone u

I was going to ask you for a better explanation of How Peering Works, but I'll go read http://en.wikipedia.org/wiki/Peering [wikipedia.org] instead. I may have more questions: if you can give a better layman's explanation than Wikipedia, I expect you'd get some upmods.:D

It points out the folly when people say "Comcast/AT&T/Verizon/whomever has to pay huge upstream bandwidth costs, bandwidth isn't free y'know!", and it always gets marked as insightful. These guys are so large, bandwidth, other than physical maintenance of their physical plant, isn't a big part of their expenses.

The problem with this argument is that these guys' physical maintenance bills are significantly higher than most everybody else's. We may quibble whether this counts as an upstream bandwidth cost; it's upstream from the customer, but not from the ISP. But even in the second case, strictly speaking, peering is basically buying some of somebody else's bandwidth and paying not with money, but with some of your own bandwidth. But you still have the costs incurrent in delivering that "payment."

Yes, if we ignore the cost of running Comcast, their bandwidth is virtually free. After all, even if they stopped using bandwidth, the cost of keeping everything running wouldn't go down much, therefore the bandwidth costs almost nothing.

Not that I really disagree with you, but to be accurate Comcast isn't a Tier 1...

Still, they do own their own fiber backbone, so any traffic staying on Comcast's network is definitely very cheap for them. And in any case, as you said, the current bottleneck in most cases is their obsolete or oversubscribed headends, not their backbone or peering points.

If the bill is zero, they're not making much either. It's a money game sure enough, but if they were making money, then the article would be about their profit from peering youtube's backbone to other providers. With the introduction of the new cisco switches/routers, and if the dark fiber is in appropriate places to do so, it's entirely possible an infrastructure upgrade would permit them to do this. However, that's doubtful. Google will almost always run at a break even, I should think, opting to send more data rather than transmit data across their networks.

Granted, I'm not a major ops center manager for a Tier1, so I really can't say for sure. Making money is always nicer than losing it, I hear.

Well, the bill being zero is just speculation from the author of the article. It doesn't imply that there are no running costs providing all of that, but that the bandwidth itself could be close to zero cost if Google is directly peering with other companies (every other article previously assumes that Google is buying their bandwidth). I work at the same place where the main IXP of my country is and while I don't know the details, it's not an uncommon thing with smaller companies either. I'm quite sure there are similar contracts between ISP's and certain big media companies that rely heavily on the Internet as it just makes business sense to everyone. It would be stupid not to use that.

Hell, there are weirder peering contracts too. A good example is that of The Pirate Bay [robtex.com], which has several AS to run their site and provide stable peering. DCSnet, PRQ and other belong all under the same umbrella and by the looks of it, have been improving their contracts with other ISP's to both get TPB to be more stable and maybe also to monetarize their peering contracts with several big ISP's. Remember that they're backed up by Carl Lundström who founded Rix Telecom AB (Port80), and Google also is peering with Port80.

Even when smaller companies are doing that, it would be stupid of Google not to utilize their infrastructure. But I'm quite certain they do, they are a geek company after all, so they must know it.

One person says Google's bill is zero, because they run the infrastructure themselves.

Another person says Google's bill is not zero because they have to maintain the network.

It's all about perspectives: Do you count internal cost or not in the discussion.
Obviously it cost "something" for the infrstructure. Is it a fixed cost internally which can be minimized and absorbed or is it an external bill which can increase significanly as the business expands.

I think the point of the article is to debunk inaccurate speculations from traders who have no technical and real commercial knowledge who may be trying to trash Google's stock for short gain. Not necessarily figure out how many Washingtons Google has to shell out.

Then again, where would be the fun of slashdot if we can't go back and forth on the chicken-and-the-edd argument...

But like even the title of the story says, we are talking about bandwidth. That doesn't include infrastructure cost. Just bandwidth which, if Google indeed does peer (and not transit) with all of their network partners, are somewhat zero. Infrastructure costs are an another matter (and would still be there, as YouTube and Google would need to run their network nevertheless)

Right the take away here is not the Googles costs are low, high, or otherwise; but that they have lots of control and plently of levers to move others. One of the biggest being they are for practical perpouses a top teir carrier themeselves.

It's relative of course, but the cost of routers and maintenance is nowhere near buying the bandwidth.

Those two parts alone, true. But they have to by the right of ways from the power/telephone company (or whoever owns the properties they run their lines over) to run their fiber. They have to pay for the fiber and actually run it. They need to handle breaks and failures of the fiber. For someone the size of Youtube, it likely is much cheaper to build your own infrastructure than buying the bandwidth dir

but the cost of routers and maintenance is nowhere near buying the bandwidth.

Here are some pics of some of Googles hardware. These are a few years old. The power interface is entirely foreign to me.
When I uploaded them to photobucket they were resized and I've since lost the originals, but, if you zoom in close enough you can see that the powersupply has a part number printed on it that includes the word 'GOOGLE', and, the ram also has chips that are individually labeled Google.
Does anyone care to explain to me how it is possible that doing such a thing is more cost effective th

They probably did purchase stuff which is already on the market in bulk. They just asked for it to be labelled Google, so people would be less likely to steal it. Although it's rare to double-sided double height sticks these days -- they must have an awful lot of RAM in each server. Perhaps the modules are actually specially made for Google. I bet the chips themselves are bog standard apart from the label though.

Here are some pics of some of Googles hardware. These are a few years old. The power interface is entirely foreign to me.When I uploaded them to photobucket they were resized and I've since lost the originals, but, if you zoom in close enough you can see that the powersupply has a part number printed on it that includes the word 'GOOGLE', and, the ram also has chips that are individually labeled Google.Does anyone care to explain to me how it is possible that doing such a thing is more cost effective than just purchasing stuff already on the market in bulk? I've been wondering it for years after seeing this.

If you're willing to buy a LOT of stuff, parts manufacturers are willing ot customize. (The threshold for "lots" varies).

Intel will sell you a custom spec'd chip if you wanted - only restrictions are it has to be based on a current production model. So if you want an i7 without 64-bit and VT, buy enough chips and Intel will provide it. Hell, if you're Google, they'll probably laser etch Google on it, too.

Power supplies - ditto. Google uses a special arrangement too, so they're probably custom-made. Which is trivial for a power supply company (as they already have lines set up to do custom builds, since 99% of their business is custom power supplies for all sorts of devices).

RAM - buy enough, and the manufacturer can do anything. Laptops often come with "custom" RAM from the OEM (usually just a label slapped on the stick). Given Google's order size, I'm sure the assembler can put GOogle on them. Heck, Apple got custom-manufactured RAM too (Mac Pro FB-DIMMs are custom made to have larger heatsinks).

And yes, Google can order in bulk, but since few can supply the order directly, Google just buys direct - cut out some middlemen, and get customization ability.

Heck, Google might get a custom motherboard too - sure it's based on an existing design, but configured to Google's specs.

Maybe Google servers as in search, but I think YouTube needs different kind of build as they're delivering massive amount of video content instead of doing database queries. I suspect the article is about Google's search servers.

Power supply chassis delivers 13.65VDC. It charges the lead acid batteries you see off to the side. Motherboard has switching converters that go straight from battery voltage to 3.3V/5V/12V, and whatever volts the CPU takes (1.65V?).

The motherboard in the system you picture is a variant of a Gigabyte GA-9IVDP. I think that system design is at least a few years old now, it wouldn't surprise me if some Google plants still have a buttload of them, but the design is continually evolving to use whatever commodit

somewhere is a rather long explanation about their hardware - on youtube ^^
gigabyte built customized mainboards for them. optimized for speed, and either a lot of ram or a lot of harddisks - and no graphics, sound or other silly things. they do UPS right on the board (top boxes in picture 2) and yes, i think it's cheaper. they might fail more often, but they can repair and reuse them!! everything is their hands and they have full control!

Laying fiber is a fixed cost for the most part, assuming the occasional idiot digging where they shouldn't. The cost of the fiber was paid for a while back, so it's a moot point.

Now that the infrastructure is out of the way, lets concentrate on the rest.

They (Google/YouTube) needs to pay network admins/engineers to maintain and also the cost of the routers. Ohh...wait.. Google has F'N HUGE data centers, so a router or two on the edge that connects to fiber that they own is a near 0 cost since they probably

That's not correct. Nobody owns all of their own fiber, no matter how big they are. Not even the Tier 1's. They may own some of it, but the majority is leased. In fact, when you purchase fiber between two locations, even though you purchase it from one party, they may in turn own some segments, while leasing various segments from multiple different providers. Even though the bulk of the costs of fiber are one time, the high capital costs result in a leasing model to cover the costs over a long period of tim

I thought that peering was largely about "you carry my bits and I'll carry yours and they'll roughly balance out". I know that this isn't like physical mail, and that the fiber can be tuned to specialize the bits going in one direction, but I'd have thought that this wouldn't be like your bog-standard interconnect between Level 1 peers.

If I'm reading your post aright, it's because they're using the other bandw

Agreed.. even the place I work with peers directly with Google (and thus by extension YouTube).. it saves us from having to spend commodity-internet dollars on traffic to Google, improves the customer experience since it's fewer hops, and we can leverage more direct control (which means less finger pointing when/if something breaks.. either its us or google.. we don't have to worry about if our upstream peers are broken); and of course it works the other way around too.

Do you think they would have run all that fiber if they weren't expecting to host a lot of bandwidth-intensive services?

In any case, they're paying for the service. If I own a trash company and give somebody free trash delivery in exchange for free doctor's visits that doesn't mean that my doctor's visits didn't cost me something.

But it is not like bandwidth itself costs money. It is always the infrastructure that you are paying for. You can either outsource network building and maintenance to a third party, or you can do it yourself.

It is no different than Google having lawyers and accountants on staff, while smaller companies only hire those people when needed. It is more cost effective for the smaller businesses to only pay for what they use, but larger companies are not bound by those same limitations. I am sure that Youtube doe

With an equally poor logic, if you hire a developer to work on project A, that person can also work in parallel on project B, because it doesn't cost anything!

The "cost" of course has to be divided up based on usage. And if they had excess bandwidth they didn't need (because youtube ate so much of what they would otherwise have), they could have sold it to someone else, making money on it. Money they are now losing.

GP is correct, the conclusion is crap.They may be saving a lot of money, but that's a whole d

It's not so much the cost to run their own fiber (marginal cost), which could be very low. The relevant cost here is opportunity cost; they could be charging other content providers to use that fiber and the revenue they're giving up is the real cost of using it for their own content.

There's a reason the concepts of scarcity and opportunity cost are introduced in the first lecture of every Econ 101 course that I know of. Too bad the concepts don't stick!

Exactly. At the point of investment in all its fiber, routers, datacenters, etc Google made a choice to invest a very large amount of money that could support something as enormous as Youtube without additional bandwidth costs. Whether that is a good investment or not depends on the ROI of having Youtube as opposed to NOT having Youtube.

So if you go back in time and DON'T invest in all that fiber, routers, etc then that money would have been available for other uses AND Youtube bandwidth costs would th

The whole point of the article is that the bandwidth *itself* has no cost. It's not irrelevant when you consider that most companies are paying by bandwidth usage and not a flat connection rate like a home user will.

No matter how much or little bandwidth they use, operational and infrastructure costs remains fixed.

If I run a shop that trades spare left shoes for spare right shoes, I don't come out even, I'm down the cost of running the shop.

But we're not talking about a trade that disadvantages either or both parties. Capacity is X. Usage is Y. While Y < X, cost remains fixed for both parties. By trading what would otherwise

You guys are looking at this from a completely different angle than a business person would. Google has that fiber REGARDLESS of YouTube's existence. It has that fiber to run its core business, advertising. Therefore the cost of maintaining the fiber is a cost to Google's advertising business. Furthermore, the cost of laying the fiber has (likely) already been paid and is no longer considered a cost but a capital investment.

Therefore, since the YouTube division is not paying for the fiber to be laid and is not paying for the fiber to be maintained, YouTube could have $0 bandwidth cost to Google.

Sorry to reply to myself but there is only an opportunity cost in using this bandwidth if the bandwidth would otherwise be used. If they are not at capacity along their fiber then there should be no opportunity cost either.

You're right, I'm looking at it from the angle of someone with half a brain.

Capital investment is not free. Laying fiber is not free. But let's pretend that we're retarded, and ignore that just for the moment. Maybe magic pixies laid it for them.

Maintaining fiber is not free. Cables get cut, hardware fails, you have to pay for electricity, and for people to maintain it. You think that's free? You don't want to pay? Fine, your hardware fails, your power gets shut off, your admins go to work for A

But that's my point. The capital investment is already made! It's therefore no longer considered a cost, but an asset. Yes they have to perform maintenance but they are maintaining it regardless of YouTube. They are maintaining it for Google's core business.

Perhaps you think its a retarded way to look at the situation but that's exactly how an accountant would look at it.

Once again, the Slashdot title has got it wrong. TFA doesn't say that Google's overall cost for bandwidth is zero, simply that their transit [wikipedia.org] costs are near zero, which specifically refers money paid to a network provider to carry your traffic.

So... am I missing something or doesn't this mean it's still costing them? Okay they're paying for it with a trade of services instead of money, but that's still bandwidth they could have sold elsewhere if they didn't have to worry about Youtube, so it's still effective cost them the amount that they offset with the ISPs, which could conceivably be close to the amount in the original report.

The question is, though: To whom would they be selling those gobs of bandwidth? The nature of bandwidth, overall, remains geographically fixed; you can't sell (much) of your bandwidth capacity in the united states to a company in Japan; they still need the pipes going, overall, from Point A to Customer B.

At the volumes in which they are dealing with, they don't really have a lot of customers who can conceivably use that much bandwidth. So it's definitely in their best interests to trade with them preferentially.

If the options are A) Trade to defer costs, or B) Try to sell to others and discover nobody else wants to buy a tenth of our capacity, they'll usually find that A) is a smarter business decision.

The question is, though: To whom would they be selling those gobs of bandwidth?

The people who would be running a net bandwidth deficit on Google's lines if it werent for the youtube bandwidth being piped over theirs.

Everyone else is not saying "sure google, no cost buddy! you rock!".. they are saying "You seem to be sending us as much as we are sending you".. remove youtube from the equation and its "holy crap we owe google a lot of money"

Owning and maintaining all that fiber is costing Google money. Even if they are not paying anything to other providers for handling YouTube traffic it is using bandwidth on their own fiber that they could otherwise sell or use for something else.

Equipment costs money to maintain. People to service equipment costs money. People and equipment to oversee and administer the network costs money. If someone accidentally severs a line in the middle of nowhere, that costs money to fix. Fiber might cost less to maintain than other technologies but that doesn't mean the cost is 0.

Are you assuming line servers are a constant maintenance fee? Although I'm sure over a year or something maybe you can calculate that out that's definitely nitpicking at best. The cost is near enough to zero that it can be considered negligible.

Google does not "rent" their fiber network. They own the fiber network. As the owner, they have to maintain the network. Given the fact that the fiber network links their data centers from all over the world, physically the fiber network covers a huge area. Any and all equipment that monitors and maintains the network has a cost. Any and all personnel to maintain that network has a cost. If they merely rented the capacity from someone then their maintenance cost is nearly 0. But they didn't rent.

This only matters once they reach capacity. If the bandwidth now being used by YouTube was otherwise going to be idle, then they would gain nothing by not using it. Since they already have the fiber in place, and since at least some of it is idle, and since AFAIK it doesn't cost any more to use bandwidth on the lines they already have, have paid for, and are already maintaining, I would think their cost should be very close to 0.

More or less in a peering situation, each provider agrees to cover their own costs. This includes laying the fiber, buying the high end routers needed for it, powering and maintaining all that, having the staff to handle problems, etc. Since each side is paying their own costs, and both have roughly equal amounts of data the other wants, it makes sense not to charge each other.

This is also why you do have to pay when you aren't one of the big guys. Your cable modem connection to your house you don't directl

nonetheless, bandwidth which tey could sell is potential money in, assuming that they can find a buyer. This arrangement guarantees no money out which they weren't going to spend anyway. To that extent, it's free.

The Wired article is from last fall. Arbor's blog post this week [arbornetworks.com] by Labovitz has better information. The most interesting data is a chart showing how 60 percent of Google's traffic takes advantage of direct peering, up from 40 percent a year earlier. Given the volume of traffic, we're talking about, there's some meaningful economics in that change.

More accurately, this is like saying "I don't own a car, so my petrol costs are zero", and everyone in the comments going "But that doesn't include your bus tickets or the time you spend walking!", and completely missing the point.

analogy (-nl-j)n. pl. analogies1.a. Similarity in some respects between things that are otherwise dissimilar.b. A comparison based on such similarity

It is used to illuminate the similarities between two items. What the Parent was saying was that the Grand Parent's analogy was inaccurate - he was comparing apples and oranges, so to speak (another analogy).

See how it works? He wasn't making fun of anybody, he was making a better analogy, and was modded up for it.

In this case, Google does not pay for bandwidth traffic across other provider's networks. That's the point of the summary and the article.

How about, I've got a subcompact that's easy to drive into town and park, and my neighbour has a pick-up truck that's great for hauling stuff from the garden store and lumber yard.

When my neighbour borrows my car to go into town, I don't charge him gas. In return, though, he lets me borrow his truck to go to the lumber yard, and he doesn't charge me for gas. Some months I do a lot of gardening and use a bit more of his gas. Other months he has a lot of paperwork to file in town and uses a bit more of my

It's more like you own a truck, and you are driving accost town anyway, so your friend asks you to pick up a box and deliver it. Sure you had to burn a little more gas, and it took about fifteen minutes more, but it's your friend, and compared to what you were doing to start with it's not a big deal, aka 'free'.

The same thing works with Google and YouTube. Compared to the whole cost of running Google, the cost of YouTube is little more than a rounding error, and odds are it is comfortably hosted in 'extra' space and run on 'extra' bandwidth that isn't needed right now, but has been paid for already, so it's basally 'free'.

I don't think that's quite it. The deal here (as best as I can tell) is that Google is sharing their pipes with other providers, making a peering agreement. Google doesn't own pipes all the way to your house, so they have to pay other providers to take the huge amount of traffic coming from their pipes and continue the journey. If Google had no pipes what so ever, they would have to pay for every single bit they send out.

But Google does have pipes, they bought up huge swaths of that dark fiber that was gett

Almost all companies lease their offices. They could buy them and save rent. It would possibly be cheaper. They don;t though. They don't want all that capital tied up in property. They can use it for business expansion instead.

So Google owns a bunch of fibre. This has a capital cost. That's money that could have been invested somewhere else, so it's not free. They could have leased the fibre from a third party. Presumably they worked out that it would be cheaper not to do this. They could probably have saved money by leasing bandwidth from a third party. The third party would then be able to amortise the costs over several customers if there's surplus bandwidth. Having capital tied up like this isn't "free".

You are correct on so many levels then wrong on so many others. Every company has to pay to maintain their own network. It just happens that Google's network is massive and goes across the country as they share huge amounts of information between data centers. It costs them significant money to maintain, but hey every company has to do that at least in their buildings. Now when they want to hookup to the internet, like most other companies they should have to pay their provider(s) for the connections. They

Almost all companies lease their offices. They could buy them and save rent. It would possibly be cheaper. They don;t though. They don't want all that capital tied up in property. They can use it for business expansion instead.

Mega corporations like Intel, Microsoft, ang Google with significant real estate needs do not lease their main campuses. They buy them and build them out the way they see fit. It also comes with local government incentives so it makes a lot of sense actually for them to own instead of to lease. Occassionally they sell the properties (like HP did) but in general they still own then property. Same goes for Walmart. I cannot recall a single store now in United States where Walmart does not own the proper

I suppose that depends. If a company is large enough that it has the money it needs to both expand the business, *and* buy the buildings/land it occupies, then it probably makes sense to buy. Why? Because, in the end, when you rent/lease, you are still paying the mortgage, taxes, and upkeep. It's just that you pay the money in the form of monthly (or quarterly, whatever) rent checks you pay to your landlord. Whether you call your monthly payment a 'leas

If ISPs are willing to give Google half a billion dollars a year of traffic in exchange for Google giving them some equivalent value of traffic on its own fiber, we should at least consider the possibility that Google could otherwise sell that traffic. Our best guess for the opportunity cost might still be half a billion dollars.

The telcom companies seemed to have the street in my office park in a perpetual state of construction during the dot.com boom. First we'd see all the different paint mark symbols from the utilities and previous fiber. Then came the new digging. There must be a ton of dark stuff still around.

I can hear the telecom companies gearing up to spin this already. They're fond of claiming that Google "steals" their bandwidth and that they need to be able to charge Google for the "privilege" of getting access to the Telecom's users. (Of course, one wonders if the telecoms would be willing to pay Google for the "privilege" of their users being able to access Google's services.) I can just see them spinning the "Google maintains such a big network that they offset bandwidth costs via peering arrangemen

Luckily, Google's fiber infrastructure is "free" - they don't pay for right of way, to maintain the connections,oversee the network, etc...

These really silly interpretations of "analysis" by financial folks is pretty amusing, actually - I suspect the report actually said something like "ignoring the deployment and on-going costs of their infrastructure Google has essentially free internet access"...

Do they think fiber, routers, switches, networking professionals, and right-of-ways are "free"?

What do I care that Google's bandwidth costs are zero when I'm stuck with my pathetic broadband connection?

"Less than free". If they don't have to pay extra for peering then they can continue the LtF business model and can disrupt whole sectors of markets. This works out in our benefit more often than it hurts us. I for one prefer to see innovation over the same ole same ole.

It's not even interesting data. The graphs make the whole presentation worse. I felt like I was reading a grad student's thesis, and a really dry one at that.

The graphs were lifted straight from the Web interface of the Arbor Networks product mentioned. It would have been nice and more professional to export the data and make some high resolution graphics instead and to provide more illuminating labels.

Unless it's changed significantly since I last saw it a little over a year ago, those graphs are not lifted straight from the web interface.

I haven't seen it in years, but they look exactly like what I recall. Maybe you're thinking of the enterprise product (Peakflow X) instead of the service provider product (Peakflow SP) which is mentioned?