FiOS customer discovers the limits of “unlimited” data: 77TB a month

Average 50TB a month—30,000 percent higher than average—and Verizon will call.

Yes, Virginia, there is a limit to what Verizon will let you do with FiOS' "unlimited" data plan. And a California man discovered that limit when he got a phone call from a Verizon representative wanting to know what, exactly, he was doing to create more than 50 terabytes of traffic on average per month—hitting a peak of 77TB in March alone.

"I have never heard of this happening to anyone," the 27-year-old Californian—who uses the screen name houkouonchi and would prefer not to be identified by name—wrote in a post on DSLreports.com entitled "LOL VZ called me about my bandwidth usage Gotta go Biz." "But I probably use more bandwidth than any FiOS customer in California, so I am not super surprised about this."

Curious about how one person could generate that kind of traffic, Ars reached out to houkouonchi and spoke with him via instant message. As it turns out, he's the ultimate outlier. His problem is more that he's violated Verizon's terms of service than his excessive bandwidth usage. An IT professional who manages a test lab for an Internet storage company, houkouonchi has been providing friends and family a personal VPN, video streaming, and peer-to-peer file service—running a rack of seven servers with 209TB of raw storage in his house.

Just another home network

I asked what exactly he does with that hardware to generate so much traffic. "Lots of stuff," houkouonchi replied. " I do some VPN stuff for people and Web/FTP/SFTP servers. A lot of friends and family stream stuff off me from my huge media collection. And I also do some P2P and Usenet stuff." Most of the storage space is taken up by videos and other media.

He's always had heavy storage requirements—in 2006, houkouonchi said he already had about 8TB of disk with no RAID. "In 2007 I went to 20x 1TB, then not too much later 20x 2TB, and then I started adding more disks and more chassis… but all the chassis that have a lot of hot-swap disk bays are rack-mount. And after having several rackmount machines, I finally decided to just buy a rack when I bought a house and got FiOS back in 2010."

This Verizon FiOS customer's home network generated 77TB of traffic in one month.

Here's what's in houkouonchi's personal data center, from top to bottom:

A 1u server acting as router and VPN server with 4 1.5TB disks.

A 1u testing server with two 1.5TB disks.

A 2u server—formerly a "colo box"—with eight 750GB disks

A 4u Solaris/ZFS backup machine with 24 1TB disks

Another 4u server—houkouonchi's main server with 24 2TB disks and two 3u storage expansion units, each with 15 3TB disks.

A 2u "Windows/miscellaneous" server with eight 1TB disks

Two 2u uninterruptible power supplies

Another 4u Solaris/ZFS server for backups with 24 1TB disks

That's just on premises. Houkouonchi also owns a 2U server running in a colocation facility with 12TB of disk on dual gigabit connections, "which I push quite a bit from as well. It runs game-servers and hosts what used to be the only LA SpeedTest.net server and a bunch of other stuff."

Saving a few bucks

Houkouonchi switched from dual 150 megabit business class connections to a 300 megabit downstream/65 megabit upstream residential plan in January (though he said he was getting 150 megabits upstream). "I only switched to residential simply because business pricing wasn't in line with residential anymore," he said. (Verizon prices the residential service at more than $200 a month for 300/65; business service for dual 150 megabit lines runs $340 a month, and 300 megabit service costs $259 a month.)

Since the switch, he's used nearly 200TB in bandwidth—an average of 50TB a month. However, traffic most months is in the 30TB range. That sort of bandwidth would cost thousands of dollars per month in most colocation facilities—with the 77TB peak month jumping into the tens of thousands based on pricing plans I looked at from a variety of hosting vendors.

Houkouonchi's network traffic—blue is outbound, green is inbound.

But his bandwidth hogging eventually drew the attention of Verizon's engineers, who monitor usage for signs of unusual patterns in traffic. This practice is to watch for both abuse of the network (such as spam and denial of service attacks, for example) and for violations of the FiOS terms of service. Those terms exclude the use of FiOS for "high volume purposes" and forbid customers to "host any type of server. Violation of this section may result in bandwidth restrictions on your Service or suspension or termination of your Service."

Houkouonchi got a call from a Verizon representative this week. "Basically he said that my bandwidth usage was excessive (like 30,000 percent higher than their average customer)," houkouonchi said. "[He] wanted to know WTF I was doing. I told him I have a full rack and run servers, and then he said, 'Well, that's against our ToS.' And he said I would need to switch to the business service or I would be disconnected in July. It wasn't a super long call."

"I don't mind upgrading to business if that's really all the problem is," houkouonchi told me. "It just surprises me they would bother going after people just to get them to pay a little bit more per month. I know that when I switched to GPON (FiOS' Gigabit-capable Passive Optical Network) about six months after a serving hub came up in my area—it took six months fighting with Verizon to get switched over—there was only one other guy on the GPON ports on the serving hub. A lot of people in my area go with Time-Warner because it's cheaper."

277 Reader Comments

I actually know this guy from DSLReports, both him and I have been members there for years. While he probably does a little illegal downloading like all of us, the vast majority of his traffic is indeed what he says it's for. He's not some little warez kiddie who has to download and store every single movie, tv show, and program he runs across to increase the size of his e-peen.

He says it's for usenet and sharing access to his "media collection". Which means he's a warez kiddie.

Yes I highly doubt that he is sharing 77TB of family videos.

Depends on if you classify baby-making as "family".

Either way unless he made it himself or owns a license it is pirating.

What he is doing is a bit extreme, but what's kinda stupid (imo) is you can't host a server. I wonder where they draw the line. So VZ FiOS customers can't have a private TS server or run private games? That's pretty lame.

As others have mentioned, it's really a question of scale. 77 TB in a month amounts to a constant 230 Mb/s upload. Assuming 8 hours of downtime each night for a residential customer, that amounts to 345 Mb/s. Any single person (or even household of people) would have a difficult time utilizing that much bandwidth unless they have servers that are publicly accessible and advertised.

Eh I don't know about 10's of thousands/month. You can get a full rack in a DC with wide open 100Mbit commit on gig uplinks for <2000/month. His 95% percent is probably not much over 100Mbit, and even if it is it's only about $5/mbit above that.

I think you'd be surprised. To hit 50TB a month, you'd have to be running (assuming 2.6m seconds/month) 20MB/sec which comes out to 160Mbits. That's quite a bit. Point being - it's still a business class usage on a residential account. You want to business class usage? Switch to a business class account, and don't complain about the $350/month when you could be paying $2k/month for what you're using the service for.

Hell, I run some servers at my home, but I'm paying for a Comcast business class account - surprisingly, it's hardly any more than their residential service (100/month for 20/5 + 13static vs 60/month for 6/2) - but their customer service for business class is light years better.

Having a server is not a business-only thing. Depending on how you define server, anyone using Skype back when they used supernodes could be operating a server as is anyone hosting a multiplayer game, using a torrent client (yes, there are legal uses), Tor etc.

Seriously look at that rack, of servers. If the guy can afford that equipment and he can afford business class internet as well. Not to mention what his power bill must look like.

I think you'd be surprised, a lot depends on what kind of hardware is inside the boxes. Remember, these are all generally just file servers, it doesn't spec what the processors/boards are. Say you have a 4u with 20x2TB hot swap drives. 20 SATA drives pull on average about 6-8 watts under load each, so 150 watts. Now what if it's running an Atom-based main board which is more than capable of handling basic file service? Whole system would be pulling 200 watts max.

4.8Kw/h per day at 10 cents per kw/h = 48 cents per day, or $15.00 per month, per server. Not all that outrageous.

It doesn't sound like he was running a business however? Maybe they define business in an odd way, but despite his equipment all his functions are personal and not for making profit?

I'm actually curious about this - I guess Verizon can do whatever they want to do though. Doesn't matter if he was _really_ violating TOS or not?

Edit: With the immediate voting down, is my question obtuse? Am I missing something obvious in the article? Yes it's a ridiculous amount of data, but "unlimited" is "unlimited".

Agreed, pisses me off of such thing. If it was just a call, then it'd be expected but then anything further than accusing him of 'violation' is plain idiotic. Then what's the term unlimited for? If the call was to offer him a business grade connection to provide an excellent customer support to him, then I'd be welcomed. Not this bs. Wonder if he would move his stuff to a GFiber location and not get a single damn call. ToS my arse.

It doesn't really matter if you're running a business if what you are using it for is business-class. I mean, I wouldn't expect to run heavy industrial equipment in a residential neighborhood, even if I'm not actually using it for industrial purposes. This is pretty similar.

A power customer can't just up and get 440 three phase service in a single family residential neighborhood. Depending on the locality there might even be zoning and fire laws against it (I think the NEC/NFPA70 frowns on it).

If he upgrades to business he'll more then likely have the same physical link.

People don't like getting gamed out of more money for the same physical service.

Eventually, we will get uncorrupted lawyers and judges and jury. When that happens, all of these criminal companies will be shut down for false advertising. TOS means nothing, if the limitations are not mentioned in the advertisement. * my arse.

EPB in Chattanooga has a different way of handling this, although I haven't heard of it being actually enforced: their TOS basically lets you run anything you want, except outbound port 25. (you have to route through their servers for SMTP, which is fair... spam sucks.) But, if you exceed the average customer's use by ten times or more, they reserve the right to charge you more and/or disconnect you.

That seems basically fair; it's not ideal, but it's not terrible.

The whole data cap thing is absolute nonsense, and it shouldn't be allowed. Data is almost free, so close to free that it's hard to measure. The only time it costs is when you have too much at once. What matters with traffic is not the total amount of traffic, but the immediate demand.

This is very much like a freeway; if you've got a two lane road, it costs almost nothing for cars to use that road. But if it gets congested, you have to expand the road to more lanes, and the more you expand it, the more it costs. This is the big capital expense in networks. But once the lanes are built, they cost almost nothing to run, and traffic doesn't wear them out.

So, in this analogy, traffic caps are like telling you that you can only drive 100 miles per month, to try to deal with rush hour problems. This is completely stupid, because if you're limited to 100 miles per month, you'll for damn sure be using your 100 miles during rush hour. It does nothing to fix the actual problem, which is not having enough lanes for the traffic during peak hours. It's just a pretext for companies to charge more while providing less.

If 77TB a month is 30 thousand percent higher than the average FiOS user, isn't Verizon ripping off a ton of people that don't need fiber internet?

You don't pay for FiOS for the unlimited cap, you pay for the speeds. The unlimited cap is a nice bonus because you don't have to worry about hitting the cap and paying fees, or getting throttled.

The meg and a half downstream I get from a dedicated host on cable is pretty reasonable.The point of having speed is the ability to move large amounts of data in a reasonable time, ie speed of connection is directly proportional to how much data you will use with it. If 77TB is 30 thousand percent of the "average" user, those other people wouldn't be able to live their life on a connection that costs a third but can still maintain over a megabyte down?

The whole data cap thing is absolute nonsense, and it shouldn't be allowed. Data is almost free, so close to free that it's hard to measure. The only time it costs is when you have too much at once. What matters with traffic is not the total amount of traffic, but the immediate demand.

That's true if it stays on the same network, but once peering and transit comes into play things get more complicated.

Having a server is not a business-only thing. Depending on how you define server, anyone using Skype back when they used supernodes could be operating a server as is anyone hosting a multiplayer game, using a torrent client (yes, there are legal uses), Tor etc.

They don't care about these things. They care that you are running business class internet facing servers using their consumer packages. When you run a server that is using more bandwidth then many companies with 100-1000 employees that is kind of a bit different then a Skype session.

That's a beautiful rack setup! Someday I'd love to have the money to upgrade my home network into something like that..

It is. I have future dreams of just making a nice setup for the limited gear I do have. My 16 port TP-link 1u smart switch is just sitting at the top of a shelving unit in my basement. The server is a medium tower sitting on the shelf below with monitor, keyboard and mouse next to it with strip UPS sitting next to all that. Network printer is on the ajoining shelving unit.

Eventually I want to at least get one of those 15u top of the desk racks, get a rack mount UPS (1u probably, I don't ask for much backup power) and a rack mount case for the server. I am torn on hosting a backup server in the rack when I finally build one or locate it out in my hobbyroom/shed once I finally it and run fiber out to it (Cat5e in house, running fiber between hobbyroom/shed and indoor switch as it has two fiber slots). I am leaning toward the later as it at least provides about 120ft seperation in the event of fire/flood/theft and the point of the backup server is BACKUP. Mostly of the data, but also the hardware (in a lower spec'd platform at least) in case the server bites the bullet, that way I don't need to rush to replace it/fix it.

Eventually, we will get uncorrupted lawyers and judges and jury. When that happens, all of these criminal companies will be shut down for false advertising. TOS means nothing, if the limitations are not mentioned in the advertisement. * my arse.

With a comment like "Everyone is corrupt including the juries" I have to ask how old are you?

And how is this false advertising? And why do TOS mean nothing? Again how old are you? Have you seen an ad for RX drugs? There is not enough time in a day to run a commercial to go over all the use cases and exceptions. But get this, most adults are capable of understanding this. Verizon Fios does not hide their TOS its easily findable on their damn website. We live in a world with 7 billion people that each have their own uses of services, if you are in the US there is 300 million people plus a shit ton of businesses on top of this that each have their own use case.

Eventually, we will get uncorrupted lawyers and judges and jury. When that happens, all of these criminal companies will be shut down for false advertising. TOS means nothing, if the limitations are not mentioned in the advertisement. * my arse.

At what point in an advertisement, do you tune out? The point of advertisement is to catch your attention, to get you to invest your own time in additional information gathering to see if what they're selling is right for you. The ad's purpose is not to drown you in technical details, most of which- if they actually provided- you would probably ignore because it's just information overload. The point of the ad is not to convey the entirety of the Terms of Service. That's for you to research on your own. Some commercials last too long as it is, without having to listen to a litany of pharmaceutical-like warnings, exceptions, and disclaimers.

To get FiOS, you don't even have to leave your damn house. So it's not like you're wasting your time driving to an office somewhere just to decide you don't want it. If you don't read the TOS before you agree to the contract, that's your loss. It's there. It's available. There is an assumption being made that you are an adult of reasonable intelligence who has done a little bit of due diligence in researching the service to which you're about to commit. Ever read that line in a contract that says, "By signing here, you certify that you have read and agree to abide by the Terms of Service?"

Eventually, we will get uncorrupted lawyers and judges and jury. When that happens, all of these criminal companies will be shut down for false advertising. TOS means nothing, if the limitations are not mentioned in the advertisement. * my arse.

With a comment like "Everyone is corrupt including the juries" I have to ask how old are you?

And how is this false advertising? And why do TOS mean nothing? Again how old are you? Have you seen an ad for RX drugs? There is not enough time in a day to run a commercial to go over all the use cases and exceptions. But get this, most adults are capable of understanding this. Verizon Fios does not hide their TOS its easily findable on their damn website. We live in a world with 7 billion people that each have their own uses of services, if you are in the US there is 300 million people plus a shit ton of businesses on top of this that each have their own use case.

His response does help explain why we have law books as thick as a tree truck though. Gotta cover all those "limitations" that people keep finding in the pursuit of trying to get around the spirit of something.

Speaking as someone with a similar setup at home (not so much storage, but more processing) I guarantee that his power bill is much higher than his Verizon bill and that going from residential to business shouldn't be that big of a deal considering the other costs.

Excellent point. As I was reading the story I kept wondering when the interviewer was going to get to the part about the electricity bill for not only the servers but also the air conditioning.

Also shows how much they can handle through their pipes, by doing nothing but gingerly switching him to a business plan.

This would make me very happy if I were a FiOS customer. The guy blatantly broke the TOS and used a huge amount of bandwidth, and Verizon simply called the guy up and switched him to a slightly more expensive plan that met his requirements.

That's impressive. I fully expected read that they terminated the connection or wanted $1000 a month or something.

My parents got one of those calls in 2001 from Time-Warner after I did 70 gigs in a month.My dad just explained he had a teenage son who was moving out in a Month and the problem would go away. It's nice to see it takes several orders of magnitude more this decade.

If the advertisement says you get A. But the TOS says you get B. Then yes, of course that is wrong as you are not getting what you paid for.

Are there seriously people out there that thinks it is okay not to get what you paid for? Really?

What are you paying for?

For my Internet? That is outside this discussion.

To provide an analogy to help people understand

1. You go to the store and pick up some Oranges.2. At the cashier, she replaces the oranges with apples.

In this case, the guy paid for "unlimited" data. Verizon decided to change things up and say, nope, you have a cap. The "advertisement space limitation" does not apply when the difference provides you with a different service.

If the advertisement says you get A. But the TOS says you get B. Then yes, of course that is wrong as you are not getting what you paid for.

Are there seriously people out there that thinks it is okay not to get what you paid for? Really?

They are advertising A. The TOS says you get A, but only if you agree to the conditions contained therein. There is either A or nothing. You either agree to their terms and get A or you don't agree and you walk away. There is no B.

Nice setup, but you should really consider consolidating that into a 25U rack, they're way more appropriate for the home (not to mention the strength of residential flooring). I've had mine for a decade and love it. If only I could get off of AT&Ts BPON and onto anyone's GPON in my area.

I wonder how much of that hardware was paid for personally and how much of it was "excess" from work.

If the advertisement says you get A. But the TOS says you get B. Then yes, of course that is wrong as you are not getting what you paid for.

Are there seriously people out there that thinks it is okay not to get what you paid for? Really?

They are advertising A. The TOS says you get A, but only if you agree to the conditions contained therein. There is either A or nothing. You either agree to their terms and get A or you don't agree and you walk away. There is no B.

Once you change A, it is no longer A. The TOS literally changes A into B. The features of A, with restrictions. It is no longer the same thing. A good analogy for this case would be:

1. You advertise Dr. Pepper. (A)2. In the TOS, you add Pepsi and Coke into the same can (B)3. You claim the drink is still Dr. Pepper.

Or more related to this story.

1. Verizon advertises service with no cap.2. TOS claims service does have a cap.

If the advertisement says you get A. But the TOS says you get B. Then yes, of course that is wrong as you are not getting what you paid for.

Are there seriously people out there that thinks it is okay not to get what you paid for? Really?

Where is this mythical advertisement which promises unlimited internet, and has no asterisk, no fine print, no caveats at all? I'd like to see it. Every time I ask to be shown this blank check, I get crickets ...

If the advertisement says you get A. But the TOS says you get B. Then yes, of course that is wrong as you are not getting what you paid for.

Are there seriously people out there that thinks it is okay not to get what you paid for? Really?

Where is this mythical advertisement which promises unlimited internet, and has no asterisk, no fine print, no caveats at all? I'd like to see it. Every time I ask to be shown this blank check, I get crickets ...

I take it you quoted the wrong person as your question seems to be to anyone who is reading your post?

To put in my two cents, as a general reader of Ars, I have yet to run across a non-corrupt company that advertises "unlimited internet". Especially when it is physically impossible to have unlimited internet.

If the advertisement says you get A. But the TOS says you get B. Then yes, of course that is wrong as you are not getting what you paid for.

Are there seriously people out there that thinks it is okay not to get what you paid for? Really?

What are you paying for?

For my Internet? That is outside this discussion.

The point, wiseass is do you understand what A is in the first place? Saying, "but I don't understand what definition of 'unlimited' you're using" just shows that you didn't do your due diligence as a consumer.