In the United States, it is usually a safe bet to attribute massive government fuck-ups to the bloated contractors we’ve outsourced our projects to.

And the electrical problems plaguing NSA’s new UT data center — described as lightening in a box that has caused $100,000 of damage each of the 10 times it has happened — do seem to stem from poorly supervised contractors.

The Army Corps of Engineers is overseeing the data center’s construction. Chief of Construction Operations, Norbert Suter said, “the cause of the electrical issues was identified by the team, and is currently being corrected by the contractor.” He said the Corps would ensure the center is “completely reliable” before handing it over to the NSA.

But another government assessment concluded the contractor’s proposed solutions fall short and the causes of eight of the failures haven’t been conclusively determined. “We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents,” said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.

[snip]

It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

[snip]

Contractors have started installing devices that insulate the power system from a failure and would reduce damage to the electrical machinery. But the fix wouldn’t prevent the failures, according to project documents and current and former officials.

Now, don’t pee your pants laughing.

But I did have two thoughts as I read this.

First, this extended confusion sounds similar to that which Iranian nuclear scientists experienced as they tried to figure out why their centrifuges kept blowing up, thanks to StuxNet. While I think the chances some kind of hack caused this are small (but not zero), I do find it ironic that we cause ourselves the same kind of havoc we cause our worst enemies.

And consider the mission!

Back in February, Keith Alexander warned of the possibility of cyberattacks on our grid (which, anonymous sources made clear, could probably only be launched by China or Russia, but that didn’t stop Alexander from suggesting Anonymous might launch such attacks). The NSA needs more authority to protect against attacks that might bring down our power sources, the head of the NSA suggested.

But the entity that proposes to wield that authority, it seems, can’t even build a brand spanking new electrical system immune from some kind of failure.

Share this entry

https://www.emptywheel.net/wp-content/uploads/2016/07/Logo-Web.png00emptywheelhttps://www.emptywheel.net/wp-content/uploads/2016/07/Logo-Web.pngemptywheel2013-10-07 20:58:112013-10-07 20:58:11How Can NSA Protect Our Power Grid from Cyberattack When It Can't Keep Its Own Power On?

It takes only one wiring mistake to create a shrapnel explosion. Miss wiring the phases from the emergency power supply for one and then “testing” it.
The Army corp of Engineers and their contractors have probably discovered a few other interesting methods as well.
As for blaming Iranians, who would have put such stuff under software control in the first place?

in ignorance, i’d bet on a serious, systemic grounding problem, perhaps associated with rebar in concrete. large-scale, high voltage wiring is both dangerous and tricky. perhaps particularly so in a building that may be laced with copper “spook” wiring to prevent the martians from learning there is unintelligent life on earth.

The idea that cyberattacks on the grid could only come from China or Russia is ludicrous. Any competent developer with the right incentives could do it. Bringing down a major portion of the grid is easier than what I do on a day-to-day basis.

particularly if it involves those little fairly simple industrial computers – i forget how they’re denominated – that rayne has written about here which control so much industrial activity here (as with, also, the iranian centrifuge computers).

For a country so paranoid about the compromising of the grid, there is absolutely no excuse to have a grid vulnerable by having IP addresses for operational hardware (I’m assuming that’s the case if they are paranoid).
The grid by definition is a well wired network that can operate independent of existing data/communications infrastructure. How difficult can it be to have separate network for communications/control of the grid by just modulating the data on the grid itself? Forget about IP addresses, just use proprietary formats.

August 2012. New Orleans. Hurricane Isaac. First real test of the $14 billion rebuilt flood protection system since Katrina. You have reporters from the New York Times and the Times-Picayune embedded in your control center, and you’re getting reports of failures in your pumps, your gates, your water level gauges, and your electrical system. Water’s rising and the public is watching. What do you do?

Answer: Turn the data off and tell the reporters everything’s working fine.

But another government assessment concluded the contractor’s proposed solutions fall short and the causes of eight of the failures haven’t been conclusively determined. “We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents,” said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.

cf the hydraulic pumps the Army Corps of Engineers installed in New Orleans:

It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

NEW ORLEANS – The Corps of Engineers acknowledged to 4 Investigates Monday that it will not have enough money to finish strengthening three New Orleans pump stations and the Sewerage and Water Board’s power plant so they can withstand Category 3 winds.

Congress and former President George W. Bush gave the Corps more than $200 million after Hurricane Katrina specifically to harden New Orleans pump stations and the old Carrollton Power Plant that runs the sewer and water system.

But in a closed-door meeting last week, the Corps told the three contractors handling the work that there isn’t enough money to retrofit the walls and roofs to withstand a Category 3′s 156-mile-per-hour winds.

Sen. David Vitter, R-La., said he found out about it from Channel 4, and he is not happy.

“Congress authorized this structure work, including this hardening work, and for the Corps to mount these cost overruns and then simply cancel the work, not even notify Congress what it will take to finish the work properly, I think, is totally irresponsible,” Vitter said.

Vitter, Sen. Mary Landrieu, D-La., and Rep. Cedric Richmond, D-New Orleans, just held a meeting last week to call for more funding for the Corps. But they are frustrated by the Corps’ constant problems with cost overruns.

After the attacks of September 11, 2001, the government’s authority to collect, keep, and share information about Americans with little or no basis to suspect wrongdoing dramatically expanded.

Except the govt approached the telecoms in February 2001 to do warrantless wiretapping, and the only one who said no was Joseph Nacchio at Qwest. So the beginning of the article (I just started reading) already implies “because terror,” because 9/11, which cannot be true.

The attacks of September 11, 2001, and the intelligence failures preceding them, sparked a call for greater government access to information.

Oh, fuck. This is stupid. Because “intelligence” backed by NSA compartmented walnut-shell paranoia is stupid. Bradley Manning was the one guy in his intelligence post in Iraq who surfed the web and tried to see a big picture, and everybody else in his unit with paranoid tunnel vision came to him to find out what was going on in the world, iirc from the trial.

I guess I should say mho. But the trouble with the paranoid IC is that nobody says mho, they have to say “I speak for all, and this is it.” At least by the time they report it anonymously and seriously to us. And if you do something that fucks their spin…

In 2006, the agency found itself running out of electricity at its Fort Meade, Md., headquarters, was unable to plug in new supercomputers and resorted to shutting down “non-essential” uses of power. That problem had been understood for at least a decade but hadn’t been prioritized high enough to fix.

The problems flared again in the summer of 2007, when NSA had to resort to rolling brownouts and delayed the deployment of data-processing equipment to keep its systems running.

At the Utah Data Center, some engineers have suggested that the best way to fix the electrical failures would be to replace the problematic parts of the system, but the idea gained little traction when project officials realized it would cost hundreds of millions of dollars, a project official said.”

“The problem, and we all know it, is that they put the appliances too close together,” a person familar with the database construction told FORBES, describing the arcs as creating “kill zones.” “They used wiring that’s not adequate to the task. We all talked about the fact that it wasn’t going to work.”

“The Utah Data Center was scheduled to be operational last month or this month — the NSA would not specify — and employ about 200 people.

The Salt Lake Tribune has previously reported that the 1 million-square-foot data center will use 65 megawatts of electricity (a single megawatt is enough to run appliances in 500 to 750 homes). Rocky Mountain Power’s largest Utah customer is Kennecott Utah Copper, which uses about 200 megawatts.”

Back in February, Keith Alexander warned of the possibility of cyberattacks on our grid (which, anonymous sources made clear, could probably only be launched by China or Russia, but that didn’t stop Alexander from suggesting Anonymous might launch such attacks).

He’s leaving out plain old American constitutional democracy and the people of the state of Utah.

Most intriguing idea I’ve heard yet — using state’s rights to stop the NSA. Utah could turn off the NSA’s water spigot.

Because of the vast nature of the Orwellian snooping machine, it takes a tremendous amount of resources to maintain it. Aside from destroying our environment by depleting our precious natural resources, this also leaves Big Brother vulnerable to decentralized political resistance.

The feds cannot force state and local governments to do their bidding. This notion is so uncontroversial that the Supreme Court has even agreed multiple times. This gives us an opportunity to hit Big Brother through legislation at the state and local levels. By stopping local and state governments from aiding and abetting the NSA’s unconstitutional behavior, we can make it impossible – or at the very least far more difficult – for them to spy on us.

In the case of the Utah data center, it is bigger than several Pentagons. Thus, it takes an astounding 1.7 million gallons of water per day to maintain the facility. Most of that water comes from a political subdivision of the state of Utah. That means we can cut the NSA off. Many other NSA centers have similar agreements with local and state governments. They can be cut off as well.

I got that from listening to Michael Boldin interview on The Scott Horton Show (the radio one, not The Other Scott Horton), who cited California Gov. Jerry Brown signing AB351 into law to refuse California’s participation in NDAA.

Earlier this week, California Governor Jerry Brown signed AB351 into law – the California Liberty Preservation Act. It makes a new state policy to refuse to help the federal government implement indefinite detention under any federal act which “purports to authorize” such.

sound of water running
Narrator: Fact. The new NSA data center in Utah requires 1.7 million gallons of water every single day to operate. Billions of Fourth Amendment violations need massive computers and the water to cool them. That water is being supplied by the state of Utah. Fact: There is absolutely nothing in the Constitution which requires your state to help the feds violate your rights. Our message to Utah: Turn – it – off. No water equals no NSA data center. Visit offnow.org.

@lysias: Or Mother Nature. All that drilling and fracking in Utah — remember what Tim DeChristopher was trying to disrupt, in Utah? What could possibly go wrong with the computer center’s critical water supply? No techies or citizen votes required. Think Fukushima. One bad day.

Think about it. One thin blonde alone in a New York apartment with two cats, a telephone and internet connection did more to ferret out financial fraud and mayhem for five straight years than the U.S. Department of Justice, Securities and Exchange Commission, Office of the Comptroller of the Currency and Federal Reserve combined. And youze guys in Washington are the ones in charge of Ahmurrica?

Michael Boldin: It passed 71-1 in the assembly, 37-0 in the state senate. I actually thought Jerry Brown was going to veto it, because it was taking a stance against the federal government. What it does is it creates a state policy that the state of California will not put any resources or participate in any way in the implementation not only of indefinite detention under NDAA, but as the wording of the bill says, “any federal law which purports to authorize indefinite detention.” So it’s future proof. And it’s a very strong first step by saying, “Hey, we’re going to say no to this, rendering that law null and void.” And we’ve seen in the past, people like Rosa Parks have proven that when you say no you can create a movement and change the world.