Navigant Research Blog

Like most forms of evil, cyber security threats do best under a cloak of silence. The fewer people who know about a threat, the more it can spread unhindered. By contrast, widespread information sharing about threats can help defenders prepare for a threat and minimize impact. But how to gather all that information in one place and get it to the people who need it?

That’s where the Industrial Control Systems Information Sharing and Analysis Center (ICS-ISAC) comes in. ISACs already exist for a number of specific industries such as information technology, financial services, and yes, even the electricity sector. But ICSs cut across many industries, such as energy, transportation, manufacturing, and utilities. These industries are served by a common set of vendors with a common set of products. So a vulnerability in one vendor’s product line could spell danger for many industries. The mission of ICS-ISAC is to spread those messages across industries.

Experts Needed

ICS-ISAC held its first conference in September in Atlanta. As cyber security conferences go, it was a breath of fresh air. Although many of the usual suspects (like me) attended, the topics were anything but the usual fare. Rather than a parade of vendor presentations, this conference was nonstop panel discussion on cyber security topics that utilities actually think about: situational awareness, workforce development, cyber insurance, establishing facility inventory, and organizational identity.

The session on building a cyber security workforce was fascinating. Schools and industry want to make cyber security a cool career choice to attract more students to the profession. Could we even entice professionals to make a mid-career change to cyber security? There is a desperate shortage of qualified cyber security experts – those who can tell a utility in practical terms what security it needs. One penetration testing firm has expanded its services from remote software testing to putting on hard hats and walking around substation yards to understand the threats facing its clients. That requires substantially more staff than running penetration tests from a remote office.

Into the Light

There were speakers from Qatar and the Czech Republic at the September conference, describing their national computer emergency response teams (CERTs). Both countries had been subjected to full-scale attacks upon their national infrastructure: Qatar in 2012 and the Czech Republic in 2013. Both have passed laws to identify their critical national infrastructure, and each now has a single response center in place to defend their infrastructures. While a large nation like the United States might require more than a single response center, the concept of having the entire national infrastructure covered by incident response is a desirable state.

The key role of ISCs centers around communication. For any organization to share the attacks it has endured, especially successful attacks, is an act of immense will. But without that sharing, the infrastructure as a whole remains in the dark. Members of ICS-ISAC are committed to break out of this protectionist mindset and share the information that will help the entire infrastructure defend itself.

The right security solutions exist and must be deployed. On top of that, let us all communicate openly so that the serious threats are exposed to the light of day before they can wreak havoc.

The precise calculation of time is at the bedrock of nearly all modern technology, including mobile telecommunications.

In technology, we’re talking really granular time – time measured in milliseconds or less. Early telecommunications were based upon time-division multiplexing (TDM), and telecoms today still depend upon successors to TDM. Newer smartphones have onboard accelerometers and gyroscopes to measure velocity in three dimensions – all time-based.

Electric grids are no exception. An instructive example is the time-synchronized phasor measurement unit (PMU). Synchrophasors are networks of PMUs that measure the phase angles of the alternating current (AC) at various points along a high-voltage network. Power flows from higher angles to lower angles, so some difference in phase angles (the phase shift) is expected, but not too much. As wide area situational awareness tools, synchrophasors can supply an early indication that something is amiss in a high-voltage transmission network. After-the-fact analysis shows that during the Great Northeast Blackout of 2003, phase angles that were normally shifted by 25 degrees had increased to a 135-degree shift. Had synchrophasors been widely available and deployed at the time, it is likely that much of the outage could have been foreseen and prevented.

Obey the Time

But time is critical to synchrophasors’ performance. To ensure coordination, all the PMUs in a network take their time stamp from a single GPS satellite. The time stamp is added to the reading and sent to a phasor data concentrator (PDC). This typically happens 30 times per second. Comparisons at the PDC or other central sites indicate whether or not phase shifts are within expected tolerances at each PMU. Out-of-tolerance measurements indicate that immediate action is required.

Here’s the problem: if the time stamp is unreliable, then a valid comparison of phase angles is impossible. Synchrophasors are but one example, and for sure there are worse things that could result from the loss of reliable time service – loss of geospatial information systems, for example. But the point remains: time is key.

All of this leads to time as an attack surface for smart grids. PMUs are one of many devices in a grid that rely upon synchronized time to give utilities control of their networks. Newer clip-on line sensors promise to make distribution management more granular as well, by taking thousands of readings per second. Again, those readings must be accurately time-stamped to be of any use.

Point of Vulnerability

Disrupt time and you disrupt the grid. How many ways are there to disrupt the time signal across a synchrophasor network? Taking out a satellite is an extreme possibility, but there are simpler earthbound approaches. My paranoid security mind just won’t let me list them in this blog however.

We depend upon time for much of what we do. We need time readings to be there for us reliably, down to the millisecond or less. And yet, time is not defined as a U.S. critical infrastructure sector. Where is the defense for this irreplaceable asset?

I must credit Frank Prautzsch of Velocity Technology Partners for raising time as an issue at a recent cyber security conference. Frank’s point, which I hope I have amplified here: while we consider complicated attack scenarios against smart grids, there are some really basic things that must also be defended. Time is among the most basic of them all.

The paper lists six threats that smart grids face. One is modernization – although, without modernization, there is no smart grid. It is a necessary evil. This section begins, “Paradoxically, modernization within the industry is also introducing new vulnerabilities.” Of course. When you replace an electromechanical device with an IT-enabled device, it’s a given that the IT threat vectors will increase substantially. As I pointed out in a recent blog, there are indeed new risks, but they are more than offset by new benefits.

The same paragraph continues by explaining that these industrial control systems (ICSs) are “often subject to periodic patches and firmware upgrades.” There is a common misconception among enterprise IT security practitioners that control systems are patched in the same way as enterprise IT systems – but that’s not the case. Many control systems have one maintenance window every 2 years, and that’s the only time they will be patched. We don’t do Black Tuesday in the control system world.

Insecure Legacy

Unisys accurately states that many existing cyber security technologies are reactive and, therefore, are useless against unknown (zero-day) attacks. However, this is not news to the ICS community, and application whitelisting and behavior-learning security tools that observe anomalous traffic have been in place for some time now.

I would also like to know if Stealth runs on its own hardware, and, if it runs in line with the control network, what kind of latency it adds to communications.

The white paper claims, “The primary reason for maintaining status quo regarding improved security is the concern that any new measure may introduce instability in highly reliable systems.” I disagree; the primary reason that the status remains quo is lack of funding. Whether that’s due to utilities being cash-strapped or security officers being unable to create a compelling business case for the funding is an open question. The second reason for the status quo is that many devices are too old to have any security onboard but still have remaining service life and aren’t going anywhere.

Wide Screen

The strongest point that Unisys makes is that the main obstacle to winning the cyber war is a patchwork strategy. This is the crux of control system cyber security. My research in the past 18 months has uncovered a marked increase in the number of utilities asking for security architectures, for a single approach to security for their control systems. Whether those architectures will translate to implementations is unclear. But at least utilities are asking to see the big picture. It would be good if Unisys offered to be part of that large-scale solution, but the conclusion of this white paper seems to say that Stealth is the solution. All security vendors can be part of the solution. None of them are the solution.

The North American Electric Reliability Corporation (NERC) is currently drafting a physical security standard for approval by the Federal Energy Regulatory Commission (FERC). This much needed proposed standard will eventually prescribe physical security for transmission stations and substations operating above 500 kV, and in some cases operating as low as 200 kV. Say hello to NERC CIP-014-1.

The stated purpose of NERC CIP-014-1 is: “To identify and protect transmission stations and transmission substations, and their associated primary control centers, that if rendered inoperable or damaged as a result of a physical attack could result in widespread instability, uncontrolled separation, or cascading within an interconnection.”

CIP-014-1, or “Sip Fourteen,” requires each transmission operator to perform an initial physical security risk assessment and periodic subsequent physical risk assessments. Effective security proceeds from a thorough risk assessment – this is the right starting place. Each risk assessment then requires an audit by a third party. The plan goes on to require operators to define risk mitigation plans, to have those plans audited by a third party, and to then implement the plan. Finally, a third party must validate that the plan has been properly implemented.

Not So Wide

This sounds like a long, drawn-out process, but it’s the right pathway: assess the risk, plan the mitigation, and then execute the plan. Each step audited by a non-affiliated third party. Security done right.

The FERC liked NERC’s proposal except for one word: widespread. Where the FERC had directed NERC to develop a plan that requires “identification of facilities whose loss could result in instability, uncontrolled separation, or cascading failures,” NERC modified the requirement to prevent widespread instability. The FERC rejected this: “The term ‘widespread’ is undefined and could potentially render the Reliability Standard unenforceable or could lead to an inadequate level of reliability by omitting facilities that are critical to the reliable operation of the Bulk-Power System.”

In other words, the FERC is nervous that any given utility may choose to define widespread instability as a total global blackout, making anything less severe outside the scope of this standard. There’s a precedent for this: the original deployment of NERC CIP standards resulted in 77% of U.S. utilities claiming that they had no critical cyber assets and were therefore automatically NERC CIP-compliant without taking any action. It’s not exactly back to the drawing board for NERC, as the FERC praised much of NERC’s proposed standard, but it is one more go-round of comments, proposals, and approval. And to the FERC: Good catch!

Plan B

One other much welcomed bit of goodness in the proposal is resiliency. The FERC writes in its comments, “Resiliency is as, or even more, important than physical security given that physical security cannot protect against all possible attacks.” Amen and hallelujah! As we learned with the Metcalf Substation in April 2013, some kinetic attacks cannot be prevented. But Pacific Gas and Electric (PG&E) had enough network resiliency in place that even the loss of a large substation resulted in not one outage. PG&E knew: you can’t hold off all the attackers, but you can have a Plan B in place to deal with their damage. And if that Plan B is automated, so much the better.