I viewed it just last night via my FiOS 50/25 connection from several different devices, and discovered that most of the time, I'm streaming at well below HD levels. (And here, I always thought my connection was working pretty darned well, too... just goes to show, it's not always easy to determine the actual resolution of fast moving video, from typical TV viewing distances.)

Maybe, If verizon gets 10000 calls an hour during next 'House Of Cards' premiere they will realize what is cheaper:Fix the problemOr keep pretending it does not exists while the need to hire 100 CSRs listening to complaints all day

#capitalism

Considering how little a telephone operator in Bangladesh costs, probably the second option.

Actually all Verizon FiOS Calls are taken by Americans in America.

Source: I used to work in a FiOS Call Center. And yes, we got calls about slow Netflix speed all the time. We were instructed to run a speedtest (on the Verizon approved and hosted speedtest site of course) and if it was in acceptable ranges we had to advise the customer that it was not Verizon's network or equipment that it was whatever provider/website(again, mostly Netflix) that was the issue.

I call bullshit. I'm a FIOS customer and I've had to make a number of service calls. Without fail, they have ALL ended up in an off-shore call center. Even in the few instances where I've used the IM help, the other end is clearly not a native English speaker. Maybe some of the higher level support is based in the US, but the initial support absolutely is not.

Read my post carefully. All calls are taken in America. I am not sure about chats since my center was calls only.

Trust there were plenty of people in just the call center I worked in(upwards of 400 CSRs at one time) who did not speak English very well. Again this was based on my experience in just one of multiple FiOS call centers.

And just to be clear, that doesn't mean that customer service is automatically better since the call centers are based in America, just trying to get some facts straightened out.

It is a tangent, but you will tend to find in-house call centres offer better service than outsourced.

In house centres can focus on quality, and can be run on a revenue neutral basis. They can take the time to ensure that staff are trained to provide the level of service they require.

Outsourced call centres tend to be paid by the call, with the price adjusted by the average customer wait time. If they badly handle your call forcing you to call back its no skin off their nose - in fact, it works out better for them if you do as they now get paid for 2 calls. Each minute they spend training is a minute that advisor isnt earning the company revenue - thats why it often seems like they don't know anything, cos noone bothered to train them as they needed the bodies on the phone to earn money.

I certainly agree and that is one of the (many) reasons I left my previous company. Verizon outsourced to my old company and the issues that you stated were common practice. There was extreme pressure to keep call times down since more calls = more $. Everything else was secondary.

There were times when floor supervisors would tell the CSRs to do just about anything, including intentionally hanging up on the customer, in order to get off a long call. Now keep in mind, this was not every supervisor, and there were(and still are) good agents that work there, but the business model of the outsourced call center is a horrible one.

To Verizon's credit, they did offer require up-training (that they would pay my company for). But it was often about new products or bundles that customers may ask about. Most of what you learn is on the job and by digging through a documentation database.

For anyone trying to blame Netflix's network for their slow performance consider this.

If Netflix itself is slow:1) No ISP will ever have the performance for 1080p video.2) The performance of other ISPs wouldn't be as high as they are.3) The performance would not suddenly go up once ISP issues are corrected.

Netflix has every reason to deliver the video to you as fast as possible because they know you want to stream these big ass movies. ISPs have every reason to spend as little as possible on their network because it's just an ongoing expense. That's why these arguments pop up in the first place: the last mile ISPs' goals directly oppose those of Internet services like Netflix. This is only going to get resolved when the goals of ISPs and Internet services are aligned, and the market is not set up that way right now. It is in other countries.

I would trust Netflix 99.99% and Verizon 0.01%.

I think that's a flawed analysis. ISPs connect to Netflix in different ways. Some have the Open Connect direct peering (because it makes business sense for that ISP), and some get content through another CDN (like the Comcast CDN that Netflix just agreed to use) or other form of paid direct peering, and some get access via an unpaid interconnect with Cogent. And even if the ISP is getting the data via the interconnect with Cogent, their port may not be as congested as the port that your (or my) ISP uses to connect to Cogent. It's possible for the performance on ISP to vary without either ISP being a particularly good or bad actor.

True, but we do know certain large ISPs are choosing to run their ports overloaded, and are refusing to correct it.

We know that Verizon had refused to correct a congestion issue with Cogent (not Netflix) without compensation. We could also say that Netflix was choosing to use an ISP (Cogent) which couldn't deliver the volume of traffic throughput requested - for business, rather than technical reasons - and was running interconnects with traffic way outside of the agreed upon ratios for interconnection.

We also know that Netflix recent struck a (paid) deal with Verizon for Verizon to directly interconnect with and deliver traffic, which should take the ISP congestion issue off the table, correct?

Quote:

Quote:

And ISPs do have an interest in delivering content efficiently and providing a decent customer experience, because if it sucks too bad customers actually will quit using it until it improves or another provider enters the market.

No, their interest is in finding that sweet spot between usable and cost. Thanks to the F-ed up system you have in the US, there are basically no other providers to go to. You guys need Local Loop Unbundling and a Government owned or government supervised company to take over the wires to offer nondiscrimintory access to retail providers

That may be true in some areas, but not all. In my area, there are three fiber-based providers, 2 of which actually serve my neighborhood - Verizon and Comcast. While they might be willing to screw over the rest of the country in areas where they don't have competition, it would seem that they would be competing tooth and claw in my region to provided a better quality service (including Netflix) to prevent me (and other like me) from dropping one to move to the other - because churn (the cost of adding and losing customers) is enemy #1 for these companies. Yet my Netflix experience has been varied, to say the least - so perhaps it isn't *only* my ISP that's the problem?

And for the record, I didn't say that ISP had an incentive to provide the best customer experience - I said they wanted to provide a decent customer experience - which is kinda the same as the sweet spot between usable and cost.

Quote:

Quote:

Netflix, on the other hand, does have an interest in shifting any blame for service problems off onto the ISPs. Perhaps the Netflix servers are just slammed and can't keep up. Or perhaps Netflix agree to use the Verizon CDN, but only at 2/3 of the service level need to ensure quality service for Verizon's customers. Look at the posts on the this tread - running about 9 to 1 against Verizon. If Netflix knows you are predisposed - and encourages you - to blame Verizon, rather than Netflix, for any service problem you have, why wouldn't they?

Difference is with Netflix, I can cancel that in a heartbeat and lose nothing, so they're more incentivised to fix things. ISPs have those contracts and exit fees to bash your head with to keep you on board just long enoguh until your anger subsides and you forget you wanted to cancel inthe first place.

If you think that you are losing nothing when you cancel Netflix, then why are you subscribing in the first place? But I take your point - Netflix doesn't have an early cancellation penalty. As for those, yes, they do discourage switching providers every couple of months. But it's a little insulting to folks to think that they just "forget" that they are dissatisfied. If anything, if they are reminded on a regular (daily) basis that the ISP service doesn't meet their expectations, and they are being held in place by a contract with a cancellation fee (which, by the way, is pro-rated from the start date of the service and becomes relatively minor by year two), I think the fact that you are forced to stay with that ISP would make you even angrier and more likely to cancel the first chance you get - you're not going to forget that life has sucked for two years but you couldn't do anything about it.

From your comments, I take it you're not in the U.S. - so tell me, do the ISPs in your land not have discounts for long-term contracts, and do they not have penalties for early cancellation?

And I'm not sure it's been demonstrated that any ISP currently are charging more to any content providers just to stream video. They certainly want to charge their customers for bandwidth/traffic volume, but I haven't seen a differential rate. And they certainly provide additional CDN-type services that support and facilitate a better streaming video experience, but that's a different issue too.

Of course, if Netflix or Level 3 or Cogent were paying Verizon for the bandwidth, then Verizon would have a very good commercial compulsion with regards to Netflix's customers.

If Netflix, Level 3, and Cogent were paying Verizon to deliver the bits that THEIR CUSTOMERS requested, than I must ask -- what are the customers paying for? As transit costs continue to plummet year over year all evidence points to this congestion being a manufactured problem.

Similarly if Verizon customers were paying metered bandwidth, the way Internet transit between networks is sold, then again they'd have an incentive for having people consume as much bandwidth as possible. I don't like the idea of metering either, but specifically selling Internet access on the premise of peak bandwidth, but not actual achieved transfer to any particular network, does certainly give some poor incentives to avoid upgrading interconnects, as you note.

Just as selling access based on "peak bandwidth" doesn't reflect the realities of traffic management, monthly caps and metered access don't either. Implementing any of these policies is a tacit confirmation that extracting money from customers' (and now content providers') wallets is the real driver.

I do find the complaints about "double-dipping" as a generic problem to be odd. Two-sided markets are extremely common, such as people who pay annual fees on credit cards, when the credit cards go around and charge merchants transaction fees.

It means nothing to say that because double-dipping happens in X, Y, and Z markets people can't object to it happening in the Internet Access market (which is vastly different). You eventually do touch on it but none of these other comparisons are at all relevant.

We commonly see two-sided markets (or "double dipping") work the other way with networks, as how medium size and smaller ISPs like Cox pay Cogent and others for Internet transit-- so in that case, Cogent gets money from Netflix to carry the traffic, and also gets money from Cox to carry the traffic. It's very unsurprising that the smaller ISPs, who are likely to have to pay Cogent and Level 3 for transit, are much more willing to accept Netflix's "Open Connect" private CDN, or indeed offer free hosting for other CDN hosts in order to reduce their transit costs.

I'm not really sure what you're trying to say here. Of course Cox has to pay someone for Internet transit, what do you think their customers are paying them for, the ability to reach other Cox customers? Cogent is selling its customers access to the Internet, regardless of whether or not the destination happens to be another customer or is reached through peers. Verizon (or another consumer ISP) charging for access TO its customers is quite different, and is a clear abuse of their market position.

Netflix has offered a great solution to help offload network congestion, and you're right it's available to any ISP that wants to take advantage of it. The sad fact is, this takes power and cooling....so the solution isn't "Free."

Given the huge volume of inbound traffic that could be eliminated, if the cost of power and cooling exceeds the savings in transit costs/needed equipment upgrades it's even more evidence of the congestion being a manufactured problem.

We could also say that Netflix was choosing to use an ISP (Cogent) which couldn't deliver the volume of traffic throughput requested - for business, rather than technical reasons - and was running interconnects with traffic way outside of the agreed upon ratios for interconnection.

I'd like to see a source for your second sentence. All of the information that I've seen about the dispute suggests that Cogent was poised to handle the traffic at the interconnect but Verizon refused to upgrade the hardware on their end. I'm not sure how you can assume anything about agreed-upon ratios since none of that is public, but I'd love a source for that too. Please tell me how ISPs that sell all of their customers asymmetrical access (while heavily marketing download speeds) expects anything other than lopsided ratios.

I don't know why they waited until after they started caving to ISPs' demands before they started doing this. They should have done this first and then just let the ISPs take the heat from their customers.

*shrug* I bet Verizon came back to them and told Netflix that the fee negotiated wasn't actually enough to "cover costs", and wanted more to ensure a "quality connection" for Netflix subscribers.

Maybe, If verizon gets 10000 calls an hour during next 'House Of Cards' premiere they will realize what is cheaper:Fix the problemOr keep pretending it does not exists while the need to hire 100 CSRs listening to complaints all day

#capitalism

Unfortunately they have already done the cost-benefit analysis and call centers in India are incredibly cheap...

Refer to the graph John Oliver displayed. Right around the time VZW and Comcast started "negotiating" with Netflix, their performance fell off a cliff. Weird how they BOTH did it at the same time too, no? It's almost as if they were colluding...

I'm sure our justice department is looking into that though, they aren't just spineless thugs who only go after petty marijuana users, right?

Make any speculation you want. I am as anti-corporate and pro an open Internet as anyone. My job is in fact working with partner ISPs on how to deliver traffic and negotiate with ISPs. My concern is that the net neutrality debate is getting skewed and the consumer is being used as the pawn. And a lot of people really don't understand how the Internet works and are being led with analogies that don't really reflect the problem. There are no "fast lanes" Traffic isn't being slowed. In fact, by Netflix working with Comcast, Vz and others on how to deliver the traffic, they are being good network citizen by moving the bulk of their traffic off the Internet pipes and to connections that better benefit the CDNs/ISPs and improve the customer experience. What makes me mad through is that Netflix is trying to pit the ISP customers against the folks that they are dependent on making their business and fear that their business model isn;t scalable as network traffic increases. Instead, they are trying to get the government to intervene on their behalf instead of doing business. IMO, not different than pitting lobbyists against our environmental issues. And to prove my point, I am against the Comcast/TWC merger. More so for content and carriage distribution but I agree that when content providers need to work with the ISP on how to move the traffic, the new Comcast will not be easy to work with.

But a much simpler solution was suggested to Comcast et al and summarily rejected by them - co-locate some servers in those ISP datacenters; that takes some of the congestion out of the equation. This is not at all hard for Comcast, Verizon and AT&T to solve, and their actions, thus far, have been to entirely degrade their consumer's collective experience in order to effectively extort money from companies such as Netflix. That is a direct result of the collusive nature of the Big Five telcos in the US and their collective oligopoly in this market.

I do, however, definitely agree with you that Comcast should be banned from merging with any company until such a time as its legal obligations towards the municipalities and the government have been met. This is true for all of the major US telcos (although AT&T are by far the most egregious example of this.)

If you read up I have already commented on this. It's not as simple as that. We are talking about managing massive amounts of streaming and bursty traffic on an IP network that has inadvertently designed not to handle this type of traffic well. Any network operator would be derelict in allowing another operator unfettered access to their network without adequate protections or the ability to control how the traffic accesses the network. And Netflix has done a very good job of calling out the network operator anytime their traffic flow is affected regardless of how it was managed or is at fault. And you are focused on only one part of the equation, you have the gateway in to a network and the last mile delivery. The servers only take care of one part of the equation.

But if you feel that this isn't a hard issue to work through, I am sure that they could use their expertise in helping them out, I can make some contacts for you. I personally know many Engineers at most of the ISPs and they have some very smart people working for them and I feel you are oversimplifying both a business and technical issue.

Also, the server model does not scale well due to the size and complexity of Comcast's network. It works better for smaller ISPs or concentrated networks.

And don't forget Verizon FiOS for franchise issues. IMO, they have been the most derelict.

And I'm not sure it's been demonstrated that any ISP currently are charging more to any content providers just to stream video. They certainly want to charge their customers for bandwidth/traffic volume, but I haven't seen a differential rate. And they certainly provide additional CDN-type services that support and facilitate a better streaming video experience, but that's a different issue too.

Content delivery costs have always been higher than standard Internet peering or transit agreements. This is because of the nature of the traffic and the costs of delivering it. This is nothing new. I find it surprising that people are so worked up over a standard business practice. To me, it's the same revelation that people are coming to that the NSA is spying on people. This is simple PR stunts with the public for Netflix to get something for free that the Internet has been charging for pretty much since its conception.

...and Verizon has filed a C&D telling Netflix to stop (or alternately to prove the source of the slowdown). If they can actually show that the content in question is being served through a direct connection to Verizon, and not transiting any other networks, it shouldn't (theoretically) be hard to back up the claim.

...and Verizon has filed a C&D telling Netflix to stop (or alternately to prove the source of the slowdown). If they can actually show that the content in question is being served through a direct connection to Verizon, and not transiting any other networks, it shouldn't (theoretically) be hard to back up the claim.

While it's true that it's probably not all that difficult to prove, given the wording and the extreme difficulty of winning a libel case in the US, Netflix could pretty much display this no matter what.

...so it's possible some of the traffic is still going through congested ports.

It's not possible that traffic is still going through congested ports unless Verizon is still deliberately routing the traffic through those congested ports. Typical network protocols route around congestion. Unless Verizon isn't following best networking practices, any traffic would route around the congested ports. The only reasons traffic wouldn't be able to route around congestion would be if Verizon is sending specific traffic through them or if all ports are congested, but that would be Verizon fault if they are all congested, and it would be a burden for Verizon to shoulder if that's the case. If Verizon cannot accommodate the service it sold to customers, then it should be at Verizon's expense to upgrade its network to actually provide the service it sold its customers.

Netflix needs to take a clue from the Youtube performance info (and my suggestions in these Ars comments over the last year) and get granular with the data. Netflix:

1) You have my zip code.2) You have my email address

EVERY MONTH you should email me a list of the ISP's in my zip code with the following subject:

Your ISP (TWC) is ranked 4th of 7 ISP's in your zip code.

Body of the message should provide more detail and a link to the web data.

BAM! Pressure the ISP's with facts.

You have 7 choices of ISPs? Where do you live and how soon can I move there?

Literal LOL. I don't. I have essentially two that matter, AT&T and TWC. Youtube's quality tool actually lists 11, but it lists a bunch of stuff that's not available in my neighborhood, much less my zip code.

That said, it also lists several wireless providers that are non-starters and AT&T is listed twice (u-verse and non u-verse), so I think a Youtube statement would say "Your ISP (TWC) is ranked 2nd of 6" if it were including all of those.

And I'm not sure it's been demonstrated that any ISP currently are charging more to any content providers just to stream video. They certainly want to charge their customers for bandwidth/traffic volume, but I haven't seen a differential rate. And they certainly provide additional CDN-type services that support and facilitate a better streaming video experience, but that's a different issue too.

Content delivery costs have always been higher than standard Internet peering or transit agreements. This is because of the nature of the traffic and the costs of delivering it. This is nothing new. I find it surprising that people are so worked up over a standard business practice. To me, it's the same revelation that people are coming to that the NSA is spying on people. This is simple PR stunts with the public for Netflix to get something for free that the Internet has been charging for pretty much since its conception.

Bollocks! Comcast, Verizon, et al. sold internet access/bandwidth to their customers. None that I am aware differentiated between the types of traffic carried. If Verizon‘s network can’t handle media traffic requested by their customers it is Verizon’s problem NOT Netflix’s. If, as a result of Verizon’s customers use of bandwith they need more money to provide the service, Verizon should then raise the rates they charge their customers.

As for it being a PR stunt can you explain why all of a sudden Verzon’s Netflix speed dropped like a rock in November 2013? It has also been noted that both Verizon and Comcast appear to be ahead of everyone else in the loss of Netflix streaming speed . . . I guess due to them both being industry leaders. A PR stunt, yes I guess it could be a PR stunt by Verizon and Comcast, to extract more money from Netflix because their customers have the audacity to want to use the bandwidth they have paid for to access Netflix.