Researchers uncover holes that open power stations to hacking

Hacks could cause power outages and don't need physical access to substations.

A pair of researchers have uncovered more than two dozen vulnerabilities in products used in critical infrastructure systems that would allow attackers to crash or hijack the servers controlling electric substations and water systems.

The vulnerabilities include some that would allow an attacker to crash or send a master server into an infinite loop, preventing operators from monitoring or controlling operations. Others would allow remote code-injection into a server, providing an opportunity for an attacker to open and close breakers at substations and cause power outages.

“Every substation is controlled by the master, which is controlled by the operator,” says researcher Chris Sistrunk who, along with Adam Crain, found vulnerabilities in the products of more than 20 vendors. “If you have control of the master, you have control of the whole system, and you can turn on and off power at will.”

The vulnerabilities are found in devices that are used for serial and network communications between servers and substations. These products have been largely overlooked as hacking risks because the security of power systems has focused only on IP communication and hasn’t considered serial communication an important or viable attack vector, Crain says. But the researchers say that breaching a power system through serial communication devices can actually be easier than attacking through the IP network since it doesn’t require bypassing layers of firewalls.

An intruder could exploit the vulnerabilities by gaining physical access to a substation—which generally are secured only with a fence and a webcam or motion-detection sensors—or by breaching the wireless radio network over which the communication passes to the server.

“If someone tries to breach the control center through the internet, they have to bypass layers of firewalls,” Crain said. “But someone could go out to a remote substation that has very little physical security and get on the network and take out hundreds of substations potentially. And they don’t necessarily have to get into the substation either.”

He points to a recent presentation at the Black Hat security conference that discussed methods for hacking wireless radio networks, which a lot of utility control systems use, including ways to crack the encryption.

“There are quite a few ways onto these networks, and utilities have to worry about this new attack vector,” Crain said.

Once in the network, an intruder can send a malformed message to the server to exploit the weakness.

“The device is supposed to throw that [malformed] message away,” says Sistrunk, “and in these cases it’s not and is causing issues.”

Neither Crain nor Sistrunk is a security researcher. Sistrunk is an electrical engineer at a major utility, but conducted the research independently of his employer and therefore asked that it not be identified. Crain recently launched a consulting firm called Automatak that focuses on industrial control systems. They began to examine the systems last April using a fuzzer that Crain created and submitted their findings to the Department of Homeland Security’s Industrial Control System-CERT, which helped them notify the vendors.

“We found vulnerabilities in virtually all implementations [of the protocol],” Sistrunk said. “Some of them are worse than others.”

Since then, ICS-CERT has published a number of advisories about the vulnerabilities, and vendors have distributed patches for nine of them, but the rest remain unpatched so far. Despite the distribution of patches, Crain and Sistrunk say that many utilities have not applied them because they’re unaware of the serious nature of the vulnerabilities.

The systems use DNP3, a protocol for serial communications that is used in almost all electrical utilities in the US and Canada to transmit communication between servers located in data centers and field devices. Electric utilities generally have a data center with two or three servers that can each monitor and communicate with a hundred or more substations, depending on the size of the utility.

The servers communicate with programmable logic controllers and remote terminal units in the field to collect status data from them in order to allow operators to monitor conditions and to allow them to trip breakers as needed or to increase or decrease the voltage.

Causing the server to crash or enter an infinite loop would blind operators to conditions in the field—something they might not initially realize since a crashed server in the data center doesn’t always register to operators, who work in other locations. Sistrunk says it would likely take operators a while to notice that the data they’re seeing on their screens, which is fed by the servers, hasn’t refreshed in a while. In the meantime, they might make bad decisions based on outdated data.

A lot of utilities also use the master servers for security purposes to control alarm systems, so crashing them would potentially disable alarms as well.

Sistrunk says a reboot of the server will generally resolve the issue, but an intruder could continue to send malicious messages to the server, causing it to crash repeatedly. He also said that in some cases, they found that the attack would corrupt the system configuration, which meant the system had to be reconfigured or restored from a backup before operations returned to normal.

Of the 25 vulnerabilities they uncovered, the most serious was the buffer overrun vulnerability that would allow someone to inject arbitrary code into the system and own the server.

One of the vulnerabilities they found exists in the source code for a popular library from Triangle Microworks. It’s not known how many vendors and products have used the library and are therefore vulnerable, but Crain and Sistrunk say that the library is one of the most popular among vendors and is used by 60 to 70 percent of them for their products.

Crain says the standard for DNP3 is not the problem but that the vulnerabilities are introduced in the insecure ways that vendors have implemented it.

The problem is exacerbated by the fact that separate security standards for how to secure power systems, which are set by the North American Electric Reliability Corporation, focus only on IP communications, overlooking the real vulnerabilities that serial communications also present.

The researchers plan to discuss their findings at the S4 security conference to be held in Florida in January.

27 Reader Comments

It seems like these types of articles have been showing up on and off for several years now about how the power grid has very poor security and is just ripe to take advantage of, and the industry keeps going yeah we know, don't worry. I am not in the industry, so they might be taking steps that I haven't noticed.

I often wonder if it will take someone crashing part of grid before any fixes are put in place. Not that I would want that to happen or suggest anyone do it, but it seems like what will end up happening before things are fixed.

I would imagine anyone who's been in the work force long enough has seen this happen in a company. Someone says, he we should fix X it's bad and gets a that will never happen, we'll worry about it later, then later comes and it's like, how come know one fixed this.

That's one of the reasons we have the NERC Critical Infrastructure Protection rules. NERC ensures that security best practices (as defined by NERC, at least) are followed, by levying amazingly expensive fines against companies who don't comply. A number of companies are often lackadaisical about compliance until their first audit, and NERC comes down with the hammer and suddenly the non-compliance company finds Jesus.

Then I get hired to come in and fix their shit.

I'd say the biggest flaw in NERC CIP is the infrequency of the updates, the low frequency of the audits, the number of ways compliance can be achieved through policy instead of through actual practical security measures, etc. The audits should also require actual pen testing.

I have a stupid EMF question: why not hardwired control commo instead of wireless? Is the interference from running parallel to high voltage too much for data lines to work along the right of way? I realize that a dedicated attacker would just tape in manually, but you'd eliminate casual hackers. (In addition to eliminating overseas threats who can't just drop by with a shovel.) Some kilovolt lines have gas pipes co-located underground, would a single data conduit buried alongside really create an arc hazard?Ah -- Perhaps I should go find the NERC guidelines, but it's just a casual question. Also, Plaid mentions policy vs. practical, so I imagine the guidelines are just that non-specific.

To be honest, NERC CIP focuses on the compliance portion more so than any true hardening. Its biggest benefit (aside from being a jobs program for IT folks) is it forces utilities who previously hooked up hardware fairly willy nilly to consider the concept of critical assets and defining perimeters for these devices. The downside is it makes automation less feasible. For the current version of the standard, serial devices are exempt from many requirements. In version 5, these devices get categorized into high medium and low criticality. Again, coming from an EE perspective, I understand putting focus on the security of these systems, but the reality is many of these devices should be air gapped anyway with external ports blocked. In reality, the biggest risk to our power infrastructure isn't a hacker coming in and shutting down a hydro unit, but rather a terrorist/angry farmer shooting at a transformer (some have up to 1 yr lead time for replacement!). Physical security should be the priority.

I have a stupid EMF question: why not hardwired control commo instead of wireless? Is the interference from running parallel to high voltage too much for data lines to work along the right of way? I realize that a dedicated attacker would just tape in manually, but you'd eliminate casual hackers. (In addition to eliminating overseas threats who can't just drop by with a shovel.) Some kilovolt lines have gas pipes co-located underground, would a single data conduit buried alongside really create an arc hazard?Ah -- Perhaps I should go find the NERC guidelines, but it's just a casual question. Also, Plaid mentions policy vs. practical, so I imagine the guidelines are just that non-specific.

Wireless (in most cases Microwave) is used only for long distance runs where running a hardwired fiber would be cost prohibitive (say a substation located hundreds of miles away in the boonies). SCADA systems are typically designed to prioritize reliability of RTU communication over latency or speed.

I have a stupid EMF question: why not hardwired control commo instead of wireless? Is the interference from running parallel to high voltage too much for data lines to work along the right of way? I realize that a dedicated attacker would just tape in manually, but you'd eliminate casual hackers. (In addition to eliminating overseas threats who can't just drop by with a shovel.) Some kilovolt lines have gas pipes co-located underground, would a single data conduit buried alongside really create an arc hazard?Ah -- Perhaps I should go find the NERC guidelines, but it's just a casual question. Also, Plaid mentions policy vs. practical, so I imagine the guidelines are just that non-specific.

A good question. I'm no electrical engineer (couldn't even be a stunt double for one) but I reckon a big part of the answer is just that wireless is a lot more convenient: a lot less planning and coordination required to plonk in a couple of wireless adapters. Most new parking meters seem to be solar powered & radio linked because although it makes the capital cost a little higher it means no trenching for cables, no disrupted public, no legal fights & compensation payments when you cut through some other company's pipe that wasn't quite where the plan showed it... And of course the EMF issues you highlight can only exacerbate all of this.

A related scenario plays out in many big companies too - some dept needs more LAN ports and IS dept (is perceived as) dragging heels, so dept solves problem with a quick Amazon shopping spree. Sometime (much) later IS dept realizes that their elegant defense-in-depth VLAN scheme is riddled with WLAN canaries still set to "password = admin"

To be honest, NERC CIP focuses on the compliance portion more so than any true hardening. Its biggest benefit (aside from being a jobs program for IT folks) is it forces utilities who previously hooked up hardware fairly willy nilly to consider the concept of critical assets and defining perimeters for these devices. The downside is it makes automation less feasible. For the current version of the standard, serial devices are exempt from many requirements. In version 5, these devices get categorized into high medium and low criticality. Again, coming from an EE perspective, I understand putting focus on the security of these systems, but the reality is many of these devices should be air gapped anyway with external ports blocked. In reality, the biggest risk to our power infrastructure isn't a hacker coming in and shutting down a hydro unit, but rather a terrorist/angry farmer shooting at a transformer (some have up to 1 yr lead time for replacement!). Physical security should be the priority.

I agree physical security is important. But consider this.

Angry farmer, pissed at the power company for taking up 100 sq feet for their transformer, will probably not shoot the transformer because there's a good chance of getting caught and prosecuted using nothing but old fashioned police work. They may have motive and opportunity but would not have the guts to execute.

A hacker working out of a basement in some country we have no extradition treaty with, leeching someone else's internet, and then going through TOR, would have motive (for the LULZ or for easy money) and opportunity and means to execute, all completely safe in their assumption that they'd get away with it.

That, the no-consequence action due to virtual immunity from prosecution, is why most hackers decide to go ahead with a hack that may do damage on a wide scale, breaking a dozen laws in the process.

There are no consequences in vast majority of computer crimes! Add to that the sheer numbers and ease of action. One hacker can perform a lot of attacks in a short amount of time, and not go to jail, and presumably continue attacking until they get bored or grow up, whichever comes first. The odds of physical vs network attacks all of a sudden look skewed toward network attacks.

Understood, but NERC CIP from my experience has been focused more on the compliance and accountability standpoint and less on the reliability portion(NERC = North American Electric Reliability Corporation). Half my day is creating change control tickets and generating documentation as opposed to fixing a problem. The impact of shutting down a critical system for upgrades(for example, the push to get XP HMIs upgraded to Win7) is more severe than for example, a domain controller failing and users being unable to access email.

I have a stupid EMF question: why not hardwired control commo instead of wireless? Is the interference from running parallel to high voltage too much for data lines to work along the right of way? I realize that a dedicated attacker would just tape in manually, but you'd eliminate casual hackers. (In addition to eliminating overseas threats who can't just drop by with a shovel.) Some kilovolt lines have gas pipes co-located underground, would a single data conduit buried alongside really create an arc hazard?Ah -- Perhaps I should go find the NERC guidelines, but it's just a casual question. Also, Plaid mentions policy vs. practical, so I imagine the guidelines are just that non-specific.

I think the problem is distance between stations and control points. Running control wires sounds nice until one looks at it close and realizes one may be substituting one set of problems for another different set of problems.

After seeing these articles over the yeas, I wonder how hard it is to actually exploit these vulnerabilities. It appears exploiting them is not that easy for someone remote to the facility such as the proverbial Chinese hacker attacking an electrical substation in Iowa. However, I wonder how close one must be physically to intercept the signals and perform the hack. Is it more like war driving for an open wifi where you physically are maybe at most a few miles away? This is not excuse the vendors, installers, or owners for not making the system as secure as possible. But is the press on whole overstating the real dangers from hackers compared to tornadoes, thunderstorms, etc..which regularly cause power outages at least locally several times a year.

"and to allow them to trip breakers as needed or to increase or decrease the voltage"

No.

1) breakers are kept closed most of the time, except for a few that are normally open.2) breakers primarily are opened by protective relays that detect short circuits. They may be opened manually to allow maintenance of equipment.3) voltage is mostly controlled automatically. some capacitors are manually controlled, but you won't bring down the grid switching them on or off.

"and to allow them to trip breakers as needed or to increase or decrease the voltage"

No.

1) breakers are kept closed most of the time, except for a few that are normally open.2) breakers primarily are opened by protective relays that detect short circuits. They may be opened manually to allow maintenance of equipment.3) voltage is mostly controlled automatically. some capacitors are manually controlled, but you won't bring down the grid switching them on or off.

It's not this cut and dried. It's true that breakers are mostly operated by protective relays, but often those relays are on a network and communicating with other devices, and those devices are able to send trip commands to the relay. This is one of the things that the DNP protocol enables, and it's what these guys are talking about when they say they can trip breakers. It's the same with capacitor bank controllers; you can remotely open some of those, and depending on the state of the grid you really might be able to cause some problems, at least locally.

It seems like these types of articles have been showing up on and off for several years now about how the power grid has very poor security and is just ripe to take advantage of, and the industry keeps going yeah we know, don't worry. I am not in the industry, so they might be taking steps that I haven't noticed.

I often wonder if it will take someone crashing part of grid before any fixes are put in place. Not that I would want that to happen or suggest anyone do it, but it seems like what will end up happening before things are fixed.

I would imagine anyone who's been in the work force long enough has seen this happen in a company. Someone says, he we should fix X it's bad and gets a that will never happen, we'll worry about it later, then later comes and it's like, how come know one fixed this.

Basically this is more scada hacking. I too am surprised the problem hasn't been fixed.

Maybe ARS can meet with EPRI in Palo Alto and the dope on why this problem isn't going away.

I have a MSEE, but this sounds like a software problem.

My recollection is trains do some sort of 900MHz track switching that has been insecure for years. I thin the fecal matter has to hit the fan before things get fixed.

so, have the Feds decided when they are going to sue these two under the CFAA? have they decided how much damage has been done to God knows how many systems and how many people are going to die because of what these two have had the audacity to do? have they been charged with deliberate attacking and hacking of public systems in order to benefit themselves?

every other report of anyone, researchers or individuals, finding vulnerabilities like these in anything have ended up having the crap sued out of them and being imprisoned. just waiting for the trial date to be released!

so, have the Feds decided when they are going to sue these two under the CFAA? have they decided how much damage has been done to God knows how many systems and how many people are going to die because of what these two have had the audacity to do? have they been charged with deliberate attacking and hacking of public systems in order to benefit themselves?

every other report of anyone, researchers or individuals, finding vulnerabilities like these in anything have ended up having the crap sued out of them and being imprisoned. just waiting for the trial date to be released!

except that they reported their work to the homeland security to allow the companies to issue updates, thus they are know to be "working on a fix" researchers,thus won't be sued. (at least in my view of the world)

Unfortunately I have a feeling that its going to take 1 serious hack, for a power station or distributor to be knocked off before this sort of stuff gets taken seriously by manufacturers and power companies...

I work for a supplier of switches, controls and fuses for the power industry, so I may be biased.

First, thing to realize is that the vast majority of the switches used for power distribution out there are not remotely controlled. The ones that are, are either small and supply only small sections of the grid, such as a neighborhood, or are very big and are monitored pretty closely. The stuff in between isn't really there yet across most of the country.

Actually, those can be remotely operated. Just order the right options when you buy your switches. Yes making those switches operate with a microprocessor is fun, and loud.

As for why wireless instead of wired control? Two reasons.

First, cost. Most situations where switches are controlled remotely are retrofits. The substation has existed for 20/30/40+ years and they want to add remote monitoring and control. It is a lot less expensive to use radios than to run hard lines.

Second, reliability. Under normal circumstances you do not need to monitor or control switches remotely. The times you do need to control them the most are during natural disasters and severe weather when the power lines are knocked down. If the power lines are down it is likely that you hardwire communications lines would be down as well. Radio lets you monitor and control things even if the wires are down. The vast majority of of the radio control systems and the switches have backup power. Batteries in the case of the controls and radios. Batteries and/or springs in the case of the switches.

Finally if you are talking about causing havoc or damage there is a much easier tool to use. It is called a bolt cutter. Most sub-stations are not manned so if you want to mess with things just walk up to the fence, break in and start operating switches. Or just take a bulldozer to things. You would be surprised how often power is interrupted by some irate guy with a piece of construction equipment.

So will cracker/hacker take out power? Yeah, it has probably already been done somewhere. Unlikely to take out really large percentages of things.

Security of the grid is looked at and improved on a continuous basis, but as well all know no matter how hard you try any large system will have plenty of holes if you dig deep enough.

Ah- thanks guys, I figured there would be practical considerations contraindicating easy fixes, since no one wants their business to be vulnerable. Unfortunately, this means that we will need a crypto solution, which is a holy grail for any number of things, not just power distribution. Or local power generation, which has its own issues.

I've done a lot of work with DNP3, but mostly in the water, waste water, and gas industries rather than electricity distribution.

I'm not surprised that there would be protocol robustness issues in DNP3 master stations - it's a surprisingly complex protocol with many optional features. (That said, the packet validity rules are as well specified as any other protocol, and someone writing a communications driver really ought to be discarding invalid packets).

On the other hand, I am surprised by the emphasis that this article places on IP versus serial traffic. DNP3 frames encapsulated in IP packets are fundamentally identical to raw DNP3 frames transmitted over RS-232 or RS-485 links. I would have expected that a DNP3 frame which causes a DNP3 driver to crash in an IP scenario would cause identical behaviour in a serial scenario, and vice versa.

Is the issue here really that lots of fuzz testing has been done against generic protocols in the TCP/IP suite (such as TCP, IP, UDP, and popular application-layer protocols such as FTP, SNMP, SMTP etc.), but little fuzz testing has been done against the DNP3 application layer prior to this work?

One point that may not be obvious to the readers is that DNP3 radio traffic is very often carried on dedicated radio networks using licenced spectrum. While it's certainly not impossible to acquire an appropriate radio for connecting to these kinds of networks, you're not going to find one at the local electronics shop. 802.11 is very rare for this kind of application. To the extent that a threat exists via this vector, it's going to come from motivated attackers with significant system knowledge and resources (Stuxnet types), not local script-kiddies. The need to obtain relatively local access to the serial networks, either physically on site or via radio, also reduces the pool of attackers for this specific vector. A nation-state attacker would need to get an agent relatively close to the assets they're attacking; the attacks described above can't be done from the comfort of your capital city on the other side of an ocean.

Anyway, I'm thrilled that serious independent white-hat auditing is now being done in this space. The vendors do seem to be getting more serious and proactive about these issues; I'm starting to see fuzz-testing certifications as a bullet-point on new product announcements.

Also we are starting to see some penetration of DNP3 Secure Authentication into this space. This is an optional part of the DNP3 protocol specification which uses cryptographic techniques to verify that master and slave devices are who they say they are, when attempting to perform sensitive actions such as remotely controlling equipment. This feature doesn't actually encrypt the commands or data values sent by the devices; its purpose is to prevent spoofing of commands or data. In SCADA applications, the Confidentiality axis of security is typically orders of magnitude less important than Integrity and Availability. However, secure authentication won't help if attackers are able to seize control of the master and its authentication keys via a malformed data packet (which might not actually require secure authentication itself).

By the way, if a crashed DNP3 server doesn't result in an alarm being raised on the operator consoles in very short order, some very bad SCADA engineering has been done in that system. I think the threat of operators making decisions based on unknowingly stale data is somewhat overstated.

While the microwave links seem like the low-hanging fruit, I'd say the physical links are the more interesting for exploits. Find the control run leading out of the fence-and-barbed-wire area, trace it out to somewhere innocuous, then dig it up. Fir it with a Raspberry Pi, data modem (a cheap phone with a USB port and the display turned off) and a lead-acid battery or two.

Do this around the country, either yourself within the life of the batteries or with some accomplices. You can now attack the grid over vast areas at the same time, making routing power around disabled areas very difficult or impossible. Or worse, cascade the failures in anticipation of known demand peaks (e.g. open breakers at the end of runs after a ramp-up, that line power isn't just going to vanish) to cause widespread physical failures.

OK, so it's a nation-state level attack rather than a 'bored hacker' or 'disgruntled ex-employee' attack, but "we don't think anyone doesn't like enough to do it" isn't the greatest mitigation strategy.

I do work in the industry and I can tell you that there many ways to distrupt it. The more complex a system the easier it is for it to fail. We have created a system with too many single points of failure that if they were to fail on their own with no outside assistance can cause wide spread problems that may take from days, months, and maybe longer to bring back online. There will be failures in the future without anybody causing them except mother nature, should she get it right sometime there will be nobody to blame except our own arogance and neglect of a well designed but neglected and over complicated system.