Entergy New Orleans, the utility that provides power to the Mercedes SuperDome in New Orleans, announced today that its technicians had determined the cause of the partial blackout during the Super Bowl last Sunday: an electrical relay the company had installed to prevent blackouts.

The relay was supposed to trip switches to redirect power in the event of a line fault over one of the cables connecting Entergy's switching gear to the stadium. In a statement, the company said that "the relay functioned without issue during a number of high-profile events—including the New Orleans Bowl, the New Orleans Saints-Carolina Panthers game, and the Sugar Bowl." But on Super Bowl Sunday, the device instead triggered when there was no fault, signaling a switch to open shortly after the second half began.

The relay has now been pulled, and Entergy is evaluating other equipment. “While some further analysis remains," said Entergy New Orleans President and CEO Charles Rice in the prepared statement, "we believe we have identified and remedied the cause of the power outage and regret the interruption that occurred during what was a showcase event for the city and state."

Update: Mike Burns, a spokesman for Entergy New Orleans, told Ars in a phone interview that the company and the relay's manufacturer, S&C Electric Company of Chicago, had conducted testing of two relays at the switching gear supplying power to the stadium. "Multiple tests conducted jointly by S&C and Entergy on the two relays installed show that one relay functioned as expected and the other relay did not," he said. "We will continue to do more testing but we believe that we have zeroed in on the device that caused the outage, and we have removed it from service."

I really dislike when companies throw those little defensive statements in. "Well it worked these other times, so we can hardly be blamed." This is not to say that I think they should be blamed, I mean things fail, I understand. I just wish they would release the statement "It happens, but we have isolated the issue and are looking into redundancy designs so that if this same issues occurs again, it will not cause a blackout." That would inspire much more confidence in me as a prospective event coordinator than the former.

Man, I don't envy those engineers. There's nothing worse than getting a nasty shit-o-gram that your relay blacked-out the Superbowl, or that your lithium ion batteries are catching fire and grounding the Dreamliner.

You might think you're being smart and adding complexity that should allow for switches to redirect power automatically in the event of an issue. But every new part you add is another potential failure point.

I'm a bit confused, if it was supposed to redirect power in case of a line fault, shouldn't the tripped system still provide power? Wasn't that the point of the switch, so that it could redirect power if the main line stopped working?

I'm a bit confused, if it was supposed to redirect power in case of a line fault, shouldn't the tripped system still provide power? Wasn't that the point of the switch, so that it could redirect power if the main line stopped working?

But wait until next year, when there's a backup to the backup to the backup system and the next outage lasts an hour.

Are stadium outages so likely that they need clever, on-site workarounds? Or has football become such big business somebody felt they had to write another check and make it even more expensive to run? Waste of time.

But wait until next year, when there's a backup to the backup to the backup system and the next outage lasts an hour.

Are stadium outages so likely that they need clever, on-site workarounds? Or has football become such big business somebody felt they had to write another check and make it even more expensive to run? Waste of time.

Any large event that gathers thousands of people in a relatively dense space needs serious infrastructure protection. Negligence in that regard can be a serious threat to public safety. Doesn't matter whether it's football, a concert, a political convention, or whatever.

I really dislike when companies throw those little defensive statements in. "Well it worked these other times, so we can hardly be blamed." This is not to say that I think they should be blamed, I mean things fail, I understand. I just wish they would release the statement "It happens, but we have isolated the issue and are looking into redundancy designs so that if this same issues occurs again, it will not cause a blackout." That would inspire much more confidence in me as a prospective event coordinator than the former.

Where did they shuffle blame? What I read was that it had functioned properly in the past, but this time did not. You added "...so we can hardly be blamed." all by your lonesome.

Where the blame shuffling comes seems to be in your interpretation. Looks to me as if they very matter-of-fact said "Our stuff broke."

The reality is that stuff does break. One moment its fine, the next it's not. AKA - Shit Happens. Not "Shit Happens, so its not our fault."

I'm a bit confused, if it was supposed to redirect power in case of a line fault, shouldn't the tripped system still provide power? Wasn't that the point of the switch, so that it could redirect power if the main line stopped working?

The description suggests that the relay operated like a circuit breaker, which is designed to protect systems from overload, rather than protecting circuits from interruption.

Simple--she put on a hell of a show. Sang and danced her guts out, and looked good doing it. And then it was over before it dragged on too long (unlike the football game).

Counterpoint: She sang and performed well. But even if that's her absolute best, some of us still found it kind of weak and her music average.

There's also the matter of Beyonce's publicist trying to get "unflattering" images taken of her during the show (the "Final Form" image macro up there is one example) pulled down off of various sites on the internet. This naturally triggered a Streisand Effect and the images are now all over the place. That's a big part of the obsession right there - we'd probably no longer be talking about her otherwise.

Back OT:

I find it rather refreshing when a corporation can just say simply "we thought it was fine, tested it repeatedly, but it broke when everyone was watching."

Simple--she put on a hell of a show. Sang and danced her guts out, and looked good doing it. And then it was over before it dragged on too long (unlike the football game).

Counterpoint: She sang and performed well. But even if that's her absolute best, some of us still found it kind of weak and her music average.

There's also the matter of Beyonce's publicist trying to get "unflattering" images taken of her during the show (the "Final Form" image macro up there is one example) pulled down off of various sites on the internet. This naturally triggered a Streisand Effect and the images are now all over the place. That's a big part of the obsession right there - we'd probably no longer be talking about her otherwise.

then it sounds like her publicist knows exactly what he's doing, and actually understands how the internet works today.

as per normal for me, i didn't watch the stuporbowl - i was out auto racing, then working in the back yard. seems we always have a race on stupidbowl sunday, and ironically, it's usually one of our best-attended events.

I'm a bit confused, if it was supposed to redirect power in case of a line fault, shouldn't the tripped system still provide power? Wasn't that the point of the switch, so that it could redirect power if the main line stopped working?

The description suggests that the relay operated like a circuit breaker, which is designed to protect systems from overload, rather than protecting circuits from interruption.

Actually, a relay provides the intelligence to operate a circuit breaker. Small circuit breakers in your house have their own tiny bit of built-in intelligence, watching for over-current. A large power system has many relays that measure many different electrical (even non-electrical) quantities and may take action based on those measurements. The most common action taken by a relay is to trip a circuit breaker.

From the article:the company said that "the relay functioned without issue during a number of high-profile events—including the New Orleans Bowl, the New Orleans Saints-Carolina Panthers game, and the Sugar Bowl."

I have to infer some of how the power distrbution is designed from the statement but I think this is how it works (just my best guess). The Stadium has at least two utility feeds from Entergy, it appears that each one of these feeds is dedicated to half the stadium (which makes some sense if the utility could only provide feeds of a particular size and the stadium needed more than one feeder could provide). One of those feeds failed leaving half the dome operating normally on utility while the other went dark except for what was supported by generator.

They say that the generators worked fine. I have to guess that the generators are only sized to power critical systems (Life Safety systems, critical cooling fans and comms, this is not unusual in public buildings).

The site is non-critical so there really isn't any need to invest in redundant systems.

So then there is the bus-tie system which will allow the systems on generator to transfer back to utility load after the feed has been reset. These have to be synchronized so that the frequency of the power is in phase with the frequency of the utility before transferring load back on (otherwise bad Bad BAD things can happen, and the whole shebang goes down again, possibly damaging all the infrastructure). I have seen this done by hand (amazing!). Mostly it is done automatically when the utility is brought up to one side of the bus and starts communicating with the generator that varies it speed very slightly to match the bus frequency.

Restoring the feeds requires coordination so you make sure you don't have anyone down the line inspecting a conductor, breaker or bus that suddenly goes live. Then there is more coordination when synchronizing the buses for the transfer (which is a hot-tie transfer because both sides are live).

Doing it in 20 minutes for a large system is still a monumental task, with substantial risk for the electricians involved. Most of your delays are related to safety protocols and communication.

You might think you're being smart and adding complexity that should allow for switches to redirect power automatically in the event of an issue. But every new part you add is another potential failure point.

In the article, it says the relay saved the day three times, then failed once. If they hadn't added that redundancy, the Super Dome would have already had three blackouts during previous games, instead of just failing this once. It's unfortunate that it had to fail during the biggest game, but you can never anticipate when faults like this will happen, and I'm sure you'd agree one blackout is better than three.

This sort of thing is pretty much standard in power grids across the world. Most major substations will have multiple redundant feeds and switchgear to automatically change over when something goes wrong. As long as your backup equipment is less likely to fail than a single power feed on its own (which it easily is), then it makes sense to have that extra complexity. It's really no different than adding a UPS to your computer. In fact, a UPS is a lot more complex and has a lot more potential points of failure than a simple electromechanical relay, but a good quality model is still less likely to fail than the power grid in most areas, so you still use it.

I really dislike when companies throw those little defensive statements in. "Well it worked these other times, so we can hardly be blamed." This is not to say that I think they should be blamed, I mean things fail, I understand. I just wish they would release the statement "It happens, but we have isolated the issue and are looking into redundancy designs so that if this same issues occurs again, it will not cause a blackout." That would inspire much more confidence in me as a prospective event coordinator than the former.

If they didn't say that their competitors would insinuate that "they didn't and don't" every time they went on a sales call. They had to get out there and frame the narrative or else their competitors would have.

it seems to me they added a single point failure into the system with that relay.

Systems are full of single point failures. What matters is what your design objectives are. For a football stadium, your overarching design objective is that you are a place of public gathering, and you have to keep all those people safe. That means redundancy and diversity in critical infrastructure. Notice that the whole dome wasn't blacked out. The system was designed fault-tolerant enough to not blackout the whole stadium when it held a capacity crowd. Everybody was able to sit and relax in their seats instead of panicking.

Secondary objective is to be a reliable place to hold football games, concerts, etc. If you can't meet your primary objective, you can't get the occupancy permit to meet your secondary objective.

Now if only the stadium had LED flood lights instead of Metal Halide, the whole US wouldn't have had to wait 15 minutes for all those banks of lights to warm up again. Metal Halide takes a long time to restart after a power loss while the bulb is still hot.

Sean Gallagher / Sean is Ars Technica's IT Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland.