The San Francisco 49ers build stadium technology in Facebook's image.

When the San Francisco 49ers' new stadium opens for the 2014 NFL season, it is quite likely to have the best publicly accessible Wi-Fi network a sports facility in this country has ever known.

The 49ers are defending NFC champions, so 68,500 fans will inevitably walk into the stadium for each game. And every single one of them will be able to connect to the wireless network, simultaneously, without any limits on uploads or downloads. Smartphones and tablets will run into the limits of their own hardware long before they hit the limits of the 49ers' wireless network.

Enlarge/ A model of Santa Clara Stadium, with a wall painting visible in the background.

Until now, stadium executives have said it's pretty much impossible to build a network that lets every single fan connect at once. They've blamed this on limits in the amount of spectrum available to Wi-Fi, despite their big budgets and the extremely sophisticated networking equipment that largesse allows them to purchase. Even if you build the network perfectly, it would choke if every fan tried to get on at once—at least according to conventional wisdom.

But the people building the 49ers' wireless network do not have conventional sports technology backgrounds. Senior IT Director Dan Williams and team CTO Kunal Malik hail from Facebook, where they spent five years building one of the world's largest and most efficient networks for the website. The same sensibilities that power large Internet businesses and content providers permeate Williams' and Malik's plan for Santa Clara Stadium, the 49ers' nearly half-finished new home.

"We see the stadium as a large data center," Williams told me when I visited the team's new digs in Santa Clara.

I had previously interviewed Williams and Malik over the phone, and that's when they first told me they planned to make Wi-Fi so ubiquitous throughout the stadium that everyone could get on at once. I had never heard of such an ambitious plan before—how could this be possible?

Today’s networks are impressive—but not unlimited

An expansive Wi-Fi network at this year's Super Bowl in the New Orleans Superdome was installed to allow as many as 30,000 fans to get online at once. This offloaded traffic from congested cellular networks and gave fans the ability to view streaming video or do other bandwidth-intensive tasks meant to enhance the in-game experience. (Don't scoff—as we've noted before, three-plus-hour NFL games contain only 11 minutes of actual game action, or a bit more if you include the time quarterbacks spend shouting directions at teammates at the line of scrimmage. There is plenty of time to fill up.)

Superdome officials felt a network allowing 30,000 simultaneous connections would be just fine, given that the previous year's Super Bowl saw only 8,260 at its peak. They were generally right, as the network performed well, even for part of the game's power outage.

The New England Patriots installed a full-stadium Wi-Fi network this past season as well. It was never used by more than 10,000 or so people simultaneously, or by more than 16,000 people over the course of a full game. "Can 70,000 people get on the network at once? The answer to that is no," said John Brams, director of hospitality and venues at the Patriots' network vendor, Enterasys. "If everyone tried to do it all at once, that's probably not going to happen."

But as more fans bring smart devices into stadiums, activities like viewing instant replays or live camera angles available only to ticket holders will become increasingly common. It'll put more people on the network at once and require bigger wireless pipes. So if Williams and Malik have their way, every single 49ers ticket holder will enjoy a wireless connection faster than any wide receiver sprinting toward the end zone.

"Is it really possible to give Wi-Fi to 68,500 fans at once?" I asked. I expected some hemming and hawing about how the 49ers will do their best and that not everyone will ever try to use the network at once anyway.

Again, absolutely not. "Within the stadium itself, there will probably be a terabit of capacity. The 68,500 will not be able to penetrate that. Our intentions in terms of Wi-Fi are to be able to provide a similar experience that you would receive with LTE services, which today is anywhere from 20 to 40 megabits per second, per user.

"The goal is to provide you with enough bandwidth that you would saturate your device before you saturate the network," Williams said. "That's what we expect to do."

Fans won't be limited by what section they're in, either. If the 49ers offer an app that allows fans to order food from their seats, or if they offer a live video streaming app, they'll be available to all fans.

"The mobile experience should not be limited to, 'Hey, because you sit in a club seat you can see a replay, but because you don't sit in a club seat you can't see a replay,'" Malik said. "That's not our philosophy. Our philosophy is to provide enhancement of the game experience to every fan." (The one exception would be mobile features designed specifically for physical features of luxury boxes or club seats that aren't available elsewhere in the stadium.)

It’s the design that counts

Current stadium Wi-Fi designs, even with hundreds of wireless access points distributed throughout a stadium, often can support only a quarter to a half of fans at once. They also often limit bandwidth for each user to prevent network slowdowns.

The Patriots offer fans a live video and instant replay app, with enough bandwidth to access video streams, upload photos to social networks, and use the Internet in general. Enterasys confirmed to Ars that the Patriots do enforce a bandwidth cap to prevent individual users from overloading the network, but Enterasys would not say exactly how big the cap is. The network has generally been a success, but some users of the Patriots app have taken to the Android app store to complain about the stadium Wi-Fi's performance.

According to Williams, most current stadium networks are limited by a fundamental problem: sub-optimal location of wireless access points.

"A typical layout is overhead, one [access point] in front of the section, one behind the section, and they point towards each other," he said. "This overhead design is widely used and provides enough coverage for those using the design."

Williams would not reveal the exact layout of the 49ers' design, perhaps to prevent the competition from catching on. How many access points will there be? "Zero to 1,500," he said in a good-natured attempt to be both informative and vague.

That potentially doubles or quadruples the typical amount of stadium access points—the Super Bowl had 700 and the Patriots have 375. But this number isn't the most important thing. "The number of access points will not give you any hint on whether the Wi-Fi is going to be great or not," Malik said. "Other factors control that."

I run a small 5 ap network and had trouble keeping the network from going to a crawl during the hurricane. I cant imagine how they will do it for a whole stadium. We use the dlink corporate ones . I also have 3 brocade ones sitting here we got for cheap that are normally $1k each. That wifi network for the 49ners will be expensive.

three-plus-hour NFL games contain only 11 minutes of actual game action, or a bit more if you include the time quarterbacks spend shouting directions at teammates at the line of scrimmage.

A little off-topic, but I've noticed that plays (in college football at least, I couldn't care less about pro) are just over 30 seconds apart typically. That makes for very nice skipping when watching a game on DVR. Watch play, skip 30 seconds, next play is set up. You can watch the entire game in less than 30 minutes usually.

This is an awfully long article to basically say, "We're building a super-awesome wireless network that will solve problems nobody has been able to solve, but all we can tell you is that we're doing all the same things any competent network designer would do."

I think the problem will not be coverage during the game, but during any delays or breaks in action. While fans are in their seats, many won't be using wifi (based on the Patriots experience). However, during halftime or during a long delay (weather, injury, replay) fans will be out of their seats, in walkways, and using wifi. I feel like the walkways should have significantly more coverage that seating areas.

Zero technical detail. Physics is physics, and building a wireless network that deals with 2.4 Ghz co-channel interference is nothing like Facebook building a data center where you can just add more bandwidth.

While there isn't a lot of meat to the article, there are a few tidbits hidden which assist in their general plan.

Excluding how the wired parts of the network work, placing access points under the stairs between sections on the way down (instead of just at the top and bottom), set to single channels (so there is little possible contention between neigbouring access points) and then tuning the transmit power on each ap (turning it down) to decrease overlap, would enable one to squeeze more access points into a smaller area and thus increase the number of total possible connections.

Yes there are issues to be resolved to deal with throttling, where to best put antennas, etc.. but it isn't impossible, just difficult.

You forgot the part where you only have 3 non-overlapping channels to work with, assuming 2.4 Ghz is still the most widely used band (like anything before an iPhone 5, Nexus 7, etc.). Unless they plan on installing a parabolic reflector under each seat, there still be co-channel interference. RF doesn't just stop propagating on a strictly defined boundary unless you build a bunch of Faraday cages, so there will be interference.

The whole point is that talk is cheap. This article is all sizzle and no steak. Unless they invented or are doing something new (which the article doesn't mention) it is as believable as your friendly neighborhood wireless vendor account manager. I am sure it will be as good as the money they throw at it will make it - no more and no less.

lyme wrote:

placing access points under the stairs between sections on the way down (instead of just at the top and bottom), set to single channels (so there is little possible contention between neigbouring access points) and then tuning the transmit power on each ap (turning it down)

Anyone else notice the iOS dominance in Gillette Stadium's WiFi network (see included screenshot of their network dashboard app), mirroring all the other webstats we get from Ars on that type of thing? Android users don't go online with their phones? It's not hard to connect to a new WiFi network on Android, is it?

It's difficult to understand the context of some statements in the article. Simultaneous association/connectivity and concurrent throughput are two very different conversations. That "conventional wisdom" is based upon real world limitations. Wireless radio frequencies, transmit and receive airtime have actual limitations. These limitations are even more apparent with 1x1 mobile devices. Some very bold statements made and promises made.

I know stadium tickets are expensive, and there's probably a fat service charge on top of the ticket prices listed. I'd just like to know if this is a premium purchase over the price of a seat.

It may be that they're touting a system that will never come close to the design usage. How many people carry wifi-only devices anymore? I know that cell data is probably shitty in a location this crowded, but the actual services their wifi provides would be interesting info to have in conjunction with this article.

While I'm not too much of a football fan, I drive by this "stadium in progress" at least once a week, and this article, along with watching the stadium coming together is generally piquing my interests.

I think the problem will not be coverage during the game, but during any delays or breaks in action. While fans are in their seats, many won't be using wifi (based on the Patriots experience). However, during halftime or during a long delay (weather, injury, replay) fans will be out of their seats, in walkways, and using wifi. I feel like the walkways should have significantly more coverage that seating areas.

If what they said was true, that shouldn't be a problem anyway. They claim that they will be able to support simultaneous use from every person in attendance at LTE speeds. While that's hard to believe, it's still unlikely that every person will be connected at once.Also, if you do a little math on the numbers he gave: terabit of throughput / 68,500 people comes out to just under 2MB/s per user.

You forgot the part where you only have 3 non-overlapping channels to work with, assuming 2.4 Ghz is still the most widely used band (like anything before an iPhone 5, Nexus 7, etc.).

While what you say is likely true, it doesn't jive with what the article discusses.

Quote:

By using somewhat unconventionally small 20MHz channels in the 5GHz range, the 49ers will be able to use about eight non-overlapping channels.

2.4Ghz is still the predominant technology. There are only 3 non-overlapping channels as colo points out. Unless the article has omitted that the stadium intends to drive towards the 5Ghz band only, the numbers don't add up. Even with 5Ghz only, I would argue that co-channel interference remains a real discussion.

Some stadiums mentioned seem to be headed the right direction. I would want to see the empirical data that supports those 49ers claims. It's interesting that the Superbowl saw less than a season game at Gillette. I noticed the only screen shot is from NetSight (Enterasys product). Was there a less technical savy crowd at the Superbowl?

Turning off channel bonding in the 5Ghz space in high density areas is not a panacea, nor is it uncommon. It makes sense when you don't need quite as much bandwidth, and you are more concerned about co-channel interference. Remember, channel bonding is a relatively recent feature for 802.11 (2009 - 802.11n). 5Ghz is great if all your devices support it, but that is a big "if" and not solvable by the infrastructure.

I am still waiting on some specifics on what they are doing that is unique. I sure it will be swell, but the technical detail is lacking.

lyme wrote:

colo wrote:

You forgot the part where you only have 3 non-overlapping channels to work with, assuming 2.4 Ghz is still the most widely used band (like anything before an iPhone 5, Nexus 7, etc.).

While what you say is likely true, it doesn't jive with what the article discusses.

Quote:

By using somewhat unconventionally small 20MHz channels in the 5GHz range, the 49ers will be able to use about eight non-overlapping channels.

As @colo says, the article is a lot talk. In fact, to me it sounds barely above a marketing hype session.

Bandwidth isn't a problem for a deployment like this - enough money tossed at it and it's not an issue. The issue is density. WiFi has a severely limited density factor to it. You can deploy a couple thousand APs in close proximity, but that doesn't mean they'll have clear access to the backhaul supporting it - to get to that, you still have to deal with a clear channel to the AP, with enough spectrum time to transmit and receive. With so few channels in the 2.4 range and so much AP density, transmission slots are going to be *HIGHLY* curtailed. They may not be doing bandwidth caps, but they'll get it anyway by nature of the spectrum usage!

The solution is a multipronged approach - utilizing apps that will minimize chatter and do a lot of localized caching, and hardcore spectrum analysis from all sorts of places around the stadium. They seem to be banking on 5 ghz channels, but that's only going to get you so far.

As for the comparisons to Facebook and a datacenter, those are all ridiculous. The only similarities they have in common are that FB and datacenters use lots of bandwidth and to isolate networks they'll be using a domain-ified network. Who cares? That's barely an issue. The problem at the stadium is WiFi connectivity. No datacenter uses WiFi for anything but low bandwidth, non-mission critical connections or for the handful of people who are in the area. If you're talking about the FB office (the FB datacenter falls under the same umbrella as any other datacenter), I seriously doubt their office infrastructure is as dense as a football stadium, and that like most of us who work at an office, we'll plug in our ethernet ports when the WiFi gets annoying.

I'd love to get a follow up article with more hard details. I understand that they are still in the planning phase and not even really in implementation but the only real detail they provided is that they seem to be banking on 5ghz devices being much more common so they can move a lot of devices there to avoid overloading 2.4ghz for devices that only support that. 5ghz wifi is inherently better for this type of deployment because it has more available channels, with even more coming eventually from the looks of the recent FCC changes, and it doesn't propagate as well through walls etc so it's easier to avoid interference.

Its great to talk the talk, but without including any details whatsoever, that's really all it is.

It sounds more like "At this time, we don't think 68,500 people will ever actually try and connect and then pull 20-40Mbps and hopefully by the time that's really necessary we'll beyond 2.4GHz restrictions."

Also 68,500 people pulling 20-40Mbps are going to saturate their devices before your 1Tbps pipe, or did I just misread that?

Why are so many people in these comments ignoring the key point in the article?

These guys essentially admit that the only reason they will be able to accomplish what they're talking about is because of the industry move to 5GHz. And that they wouldn't be able to do it if they had to deal with today's mix of cell phones (i.e. not enough people currently have 5GHz-capable phones).

That's essentially the whole story: Why will the 49ers be able to provide 100% Wi-Fi coverage when no one else has? Because it'll be a year-and-a-half from now, and technology improves over time.

The article sort of implies no one else is working on 100% coverage in their stadiums for that kind of time frame. I guess that's possible. Maybe the 49ers just have an advantage because they happen to be rolling out a whole new network at exactly the same time that 5GHz will allow for 100% coverage.

Setting up the stadium network at level 3 instead of level 2 is also sort of interesting. But face it, backhaul isn't really the constraint for serving a stadium full of people with Wi-Fi, so that's just an interesting optimization.

I'd say it's highly improbably that a stadium full of people would all be connecting to the wifi network at all, let alone all at the same instant. So these delusions of grandeur are pretty much risk-free at this point.

Its great to talk the talk, but without including any details whatsoever, that's really all it is.

It sounds more like "At this time, we don't think 68,500 people will ever actually try and connect and then pull 20-40Mbps and hopefully by the time that's really necessary we'll beyond 2.4GHz restrictions."

Also 68,500 people pulling 20-40Mbps are going to saturate their devices before your 1Tbps pipe, or did I just misread that?

I think the bandwidth numbers gloss over that there's an assumption that a big chunk of bandwidth demand will be video served off of servers at the stadium. That bandwidth never hits the external pipe.