Yet Another Edition of “You Were Warned”

You were warned about the possibility of security threats to your systems. Repeatedly–the video above is just one such warning. What’s it take to get through to you–a clue-by-four alongside the head? A massive, lengthy power outage you can’t resolve for days or weeks, with consumers calling for managements’ heads on pikes? A complete tank of your company’s stock value? The Department of Energy on your doorstep, taking possession of your site as it investigates you?

I love this part at 32:28 into the video where Ralf Langer says,

“…many things we thought about cyberwarfare earlier just were proven wrong. …”

Everything you thought you knew about infosec/cybersecurity needs to be revisited. The assumptions you’ve been using are clearly wrong.

Now get a frigging clue and revisit your security policies. STAT. You can start with checking these:

— No USB or other external media which have not been deeply screened for infection.

— External network connections to production equipment are to be avoided at all costs. Connections between corporate business and the power grid should be closed, dedicated network. Revisiting appropriateness of traditional isolation of production networks might be worthwhile.

— No third-party contractors permitted on site that do not comply completely with power company security policies, including spot inspections. (You do spot inspections, right? Contractors are screened coming in and out of facilities, right?)

It’s the only excuse I can think of as to why security measures and subsequent audits of the nation’s power grid for infections and intrusions from network and external devices haven’t removed these threats.

By the way, this 2009 document making suggestions to power companies about security measures is now out of date and needs to be revisited, in light of the Senate Intelligence Committee’s authorization of cyber weapon deployment and subsequent blowback risk, let alone the case of USB devices laden with crimeware.

Dear Fellow Americans: I really hate feeling like Cassandra. I’d love to see the power industry and our government prove me wrong by preventing outages related to security breaches about which they’ve been warned. At the rate they’re going, you’re going to end up on the short end of the stick, without electricity to read my anticipated future post which I expect to entitle, “I told you so.”

You might want to contact your government representatives and ask them what they know about power grid security and if they’ve actually done anything to investigate the safety of power in their district. If their understanding is shaped by the Department of Energy’s latency, they need to be brought up to speed and pronto. Don’t wait until you don’t have the juice to read my next post on this topic.

Shortly before Christmas, Aigner wrote a letter seeking the support of Germany’s vice chancellor, Economics Minister Philipp Rösler of the business-friendly Free Democratic Party, also a junior partner in Merkel’s coalition government. In it, she demanded that he back “the federal government’s entry into the grid operators’ market” — in other words, a partial nationalization of Germany’s power lines.

A “strong government partner” could “provide security” in connecting offshore wind farms to the German power grid, she wrote. The proposal has been floating around for some time, she noted, adding that he should examine it “again, and thoroughly.” Voters, Aigner reminded Rösler, don’t understand why they should pay higher electricity prices to cover the risks of the federal government’s shift away from nuclear power and toward green energy, while grid operators are raking in “a high, guaranteed return on their equity.”

I was thinking about this as I read your post, until I got to the “Dear DOE” section. Even so, having to clean up one entity (DOE) has got to be easier than cleaning up DOE plus dozens of other private entities.

@Peterr: It’d be as difficult to nationalize power production in this country now as it would be to migrate to single-payer health care. Corporate interests simply won’t permit it. Only way around it is to encourage every household to consider moving to personal/family off-grid production. At the rate DOE and power companies are moving on info/cyber security, it’s in our best interests to pursue small scale highly local production as a backup.

@P J Evans: Interesting–what kind of workstation? We may see more shared appliances on closed networks soon, both for cost reduction and for improved security.

The power companies should be giving holy hell to our intrepid intelligence/DoD/cyberwarfare agencies. To me, it just shows that govt. hackers are much more dangerous than the ones that our DoJ viciously pursues, hounding them to death.

It also reminds me of the horrific damage the CIA has done to immunization and other health aid programs with their “brilliant” plans to hunt down terrorists.

Did this incident happen at a regular coal fired power plant or a nuke power plant?

I don’t know of any company who has 100% compliance with computer security procedures so I think it’s just a matter of time before this happens again. I hope somebody is writing some software that runs on all of these machines monitoring for the stuxnet type viruses and trapping them before they do damage. You’d think that would have happened already. But then again, we already know that the physical security for nuke plants is lax in at least some places, so how good is their computer security?

I can remember in the early days of anti-virus software. I worked at a company and spent a lot of time in our huge computer lab. Aisle after aisle of computers, racks of computers stacked, and our QA people had to keep the anti-virus software up to date by hand, run full scans regularly, etc. It’s a lot easier now but still a horrendous job.

Nebraska’s power grid is state owned and its the most reliably Republican state in the country. We need to think bigger about D/R coalitions around fundamental issues that matter to the future of the country. Basically, ways to work around the Southeast.

The most profound thing I saw on the video was near the end, about how stupid it is to invest heavily in digital meters. Not on privacy/paranoia grounds, but on needless complexity grounds. Simple spinning meters do the job fine and employ people to walk around and check them. Moving to digital is needless anti-labor cost cutting and good business for GE, but not good for the country as a whole.

@Rayne: “Corporate interests simply won’t permit it.” Just like big banks won’t allow themselves to be regulated. And when, as a result of deregulation, they crash themselves and everybody else, the taxpayers are forced to bail them out.

@Rayne:
It was an HP machine with a network connection; most of the software lives on the server. Locally, we got Office (currently 2007) and IE (7, I think). Everything else went through the network – they have an intranet, which I hope is as secure as it should be.

Without access permission (and a dongle) you can’t get from outside into anything but e-mail and your own personal information (which I’ve done).

@fnook:
It really depends on local conditions. It’s certainly not cost-effective to have people spending eight hours a day, five days a week, reading meters by hand in either urban or rural areas. That’s why they’re going to electronic metering: it can be done by someone with a drive-by transceiver, without them having to get out and walk; it’s safer, too, since they don’t have to deal with dogs (that’s not a small matter: dog bites are a major cause of injuries for meter readers).
I doubt that most people ever even look at their meters.

@P J Evans: Should see this model increasingly with health care environments because of HIPAA regulations, reduces leakage of personal info. But it does reduce cost to operate as well in terms of equipment and licensing.

@joanneleon: The folks in government and many in corporate management do not yet grasp the concept of asymmetric warfare. Not every attack will look like a nation-state launching a weapon; some of it may be crimeware or malware that cripples productivity and undermines standard of living. Until the boneheads who make management decisions in private and public sectors alike grok this — crimeware may be cyberweaponry — they’re going to continue to treat incidents like these malware infections as annoying facts of life.

@fnook: Unfortunately, electric utilities as well as oil and gas companies have historically given heavily to Republicans. It’s still a D/R story in that respect, particularly since the current administration has made some noises about climate change-related regulation. The red state public also tends to vote conservative rather than centrist because they too readily believe the negative media that energy companies fund with regard to climate change–it’s evident in the relative dearth of environmental and climate change reporting in the U.S. compared to any other country.

When you figure out how to break this log jam, let us know. I think it’s going to take real pain before people clue in–more droughts, more food shortages, and less electricity.

I can state from personal experience that security controls can be implemented, as suggested, until someone with some political “pull” in the company whines about not having access to something or other that they do not necessarily need. Then, the concept of the isolated control network goes out the window.

As for USB devices and CD-ROM drives, they can be disabled for anyone. Usually, an administrator can gain access through his or her password.

@Sojourner:
It took two or three go-rounds before we got reliable access to some of the output of our own department – maps that had been scanned to PDFs. Not current, but if you wanted to know what had been there twenty or more years ago – it happened fairly often – you needed that access. It was a matter of getting the ‘ownership’ issue settled.

From shortly after the infamous northeast blackout of 1965 until I finally stepped off the long ramp into full retirement in the late naughts I worked for suppliers of and consultants about SCADA and energy management systems for electric utilities. After-the-fact analyses thereof identified the utilities’ internal organizational walls and their resultant cumbersome processes as one of several root causes of the blackout. Most of the vertically integrated companies subsequently combined their Generation and Transmission Departments, which previously had in most cases reported through separate channels all the way to the CEO, into Bulk Power Departments reporting to a VP. In related moves, the usually-separate 24/7 power plant dispatch and transmission control offices were combined into unified power system operations control centers with the operators of all functions within conversational range of each other the same room, and all under a single manager.

To complicate matters, these organizational changes were taking place against a background of radical technological change. The electro-mechanical supervisory control systems (predecessors of SCADA systems) that had been in use since the 1920s were beginning to be replaced by solid-state digital technology, and computers were starting to take on the closed-loop control function then called “load frequency control” (later “automatic generation control” and now known as “balance control” IIRC) previously accomplished with analog systems.

Over the subsequent quarter-century these early efforts evolved into energy management systems (EMS – not to be confused with the same three words associated with the control of building environments) that performed the supervisory control and data acquisition (SCADA) and automatic generation control (AGC) functions, as well as a variety of power system modeling for optimization, stability and planning purposes. All the while these EMS systems remained isolated from other computing systems within the utility organization. The corporate data processing departments, which were then still 8/5operations except for late shift computer operators, wanted nothing to do with having to support the 24/7 availability requirements in the control center.

It is significant that the so-called “deregulation” movement in the industry coincided with the emergence and growth of the internet, because it was the changes required by the former that drove the need to connect the EMS electronically in real time not only with other entities within the company that previously only required periodic reports, if anything, but also in real time with an expanding universe of outside entities as well, such as newly formed power marketing companies. The Internet was the path of least resistance with which to do this. Before deregulation, control centers’ connections to the outside world were limited to the power plants they controlled, the substations through which they connected to a neighboring utility companies, and point-to-point 24/7 data channels to the control centers of the adjacent utilities, and to whatever regional reliability or power pool organization they were affiliated with. One ironic result of deregulation is that it led to removal of part of the next-day-and-beyond bulk power planning function from the control centers in two separate locations. That is, it undid some of the changes deliberately made in the wake of the 1965 blackout. This was done to supposedly assure that there was a “level playing field” for the purchase and sale of bulk power.

Many of us who had been around the block a few times expressed our concerns about the direction things were taking, but Wall Street was driving the bus and more than a few power system engineers and operators double their salaries by jumping the fence over to the power marketing companies. And many a power company executive now saw himself (they were almost all men at that point) as wheeler-dealers. We tried to write language into the contracts that would require the vendors to build strong walls around the EMSs, but that’s hard to do when you need to punch so many holes through them.

To summarize, I believe it would be very difficult at this late date to completely I could isolate the control centers from the Internet. That is, unless the resources were available to set up an entirely separate private Internet that connected not only all of the of operating utility companies, but all of the power marketing entities that have emerged as well as the large industrial purchases of bulk power.

Before California’s problems hit the headlines at the turn of the millennium there was a cartoon pinned up on the bulletin board in the offices of one of my clients. It showed a full-bearded man dressed in skins and carrying a club, together with a young boy similarly attired. The man is saying “No, son, we didn’t have a nuclear war. We deregulated the electric utilities.” It stayed there for the nearly full decade I worked with that company.

@ex-PFC Chuck: Thanks for sharing that. I’ve been thinking about the re-isolation of energy production from energy business operations. It’s true that it’d be difficult to remove networking, but I believe internet exposure to production could be eliminated. It would require a one-direction push of data via network out from the plant to business, and input to the plant through a gatekeeping mechanism–buffering, if you will.

Have you looked at the 2009 DOE document (PDF) linked in the document? they offer some rough layman’s schematics. Wonder if you saw any opportunities or flaws in them, given your background.