2. Defense in depth does not obviate the need for proper risk management and addressing major risks

Project Basecamp has sensitized Digital Bond to the increasing use of defense in depth as an excuse rather than as a security principle. Now that “SCADA and DCS are not connected to other networks / air gap” excuse is slowly dying. Defense in depth has replaced it in vendor and CERT bulletins as the new excuse to avoid addressing the most significant risks in a control system.

The advice the vendors and CERTs are providing on the importance of defense in depth is useful and correct, but it typically does not address the specific risk that is the cause of the bulletin. Imagine you were having trouble seeing, and the advice was brush and floss your teeth, eat a healthy diet and exercise. Not bad advice, but not addressing the real problem. When vulnerabilities occur that greatly increase your risk, push the vendors for an actual solution to the vulnerability in place of SCADASEC 101 advice.

3. Digitalbond.com labeling entries as SCADASEC 101 or Control System IT 101

From our ten year relationship with loyal blog readers, we know that a large portion of our readers are very experienced in ICS security. I’m sure these readers groan, as I often do, when you read another article about a very basic comment. Probably 95%+ of what is written are these basic concepts rehashed over and over.

On this site we try to bring new information, new opinions, and new tools to the experienced ICS Security professional. And this will continue to be our focus.

We are pleased to announce that Michael Toecker joins Digital Bond this week as a ICS Security Consultant and Researcher. Michael previously worked at Florida Power and Light and Burns & McDonnell.

He is a bit different than the typical Digital Bond employee in that he is an engineer first and foremost and has migrated into the security world. We are excited to have this capability on the team to help better understand how vulnerabilities can be leveraged to affect the process; better understanding the impact of an attack, especially in the electric sector. We are already thinking about how to leverage this new talent in some of our ongoing research projects.

And Michael has opinions and is not shy about sharing them, as those of you who know him or follow him @mtoecker have heard and read. He will be sharing those now on the digitalbond.com site as well.

A number of loyal readers have been sending in examples of vulnerable, Internet accessible control systems. The example below from Patrick Stave of Norway is representative of what we are receiving. In this case, I 100% agree with ICS-CERT that if you have your SCADA or DCS on the Internet, you are facing an increased risk.

Check out Shodan for “NS web interface”.
This is a HMI-panel with remote operation from Omron.
Runs on 1980s Microware OS9 operating system.

Only operation mode requires authentication.
All panels where the password has been changed can still be monitored on URL /monitor.htm

Also some of theese can be altered with the engineering software CX-designer over web without authentication (of course a result of port forwarding from the user).
Also the panel can be used as a gateway to connect to the PLC and visa-versa.

Have found several examples of PLCs directly configurable / controlled over internet without authentication.

Patrick then provided some screenshots showing some displays:

British monitoring and water feed control for a hydroelectric power plant. Accessible / controllable / programmable over the Internet, with no password.

Shortly after Rubén’s vulnerability announcement concerning their Modicon Quantum line of controllers, Schneider Electric released updated firmware for their various controller’s Ethernet cards, as well as an advisory (I sarcastically love the combination of “we take security very seriously” and “hard-coded backdoor credentials”). The partial fixes are incorporated into version 5.01 of their firmware for our NOE 771 01 module, which was shown in Basecamp.

The fix is rather sad. It disables the Telnet and WindRiver debug service issues, but leaves backdoor accounts and an FTP server enabled. The WindRiver debug port is fairly well-known (Dillon Beresford wrote up some nifty documentation on automating WDB agent attacks, including showing how to remotely alter how a controller boots by using the service to overwrite the bootline). The WindRiver Telnet service is quite a bit less-known.

The telnet service is really a C interpreter. Fermi National Labs has a nifty intro to it here. Because Schneider left debugging symbols in the Quantum line of controllers, it’s incredibly easy to use the C shell: you get function names. You can type commands like ‘strlen(“foo”)’ and the shell will return 3. On systems which lack debugging symbols, you’re left having to reverse engineer the firmware to find the function location, and then calling the function by using the pointer. For example, suppose that the strlen() function was found to begin at offset 0x0016fc18 in the disassembly:

-> 0x0016fc18(“foo”)
value = 3 = 0x3

It’s pretty interesting because the command line is totally type-free C. You can treat any spot in memory as a function without having to cast it. Even if your target lacks debugging symbols, that’s okay: you can do assignments on the command-line to make your life easier and make your code cleaner. This works for both function names and your own declared variables:

You can use the Telnet console for a lot more evil things, too, of course. vxWorks includes a full bsd sockets library, meaning that you can write a C program to run as a port scanner from the telnet console. Kinda cool. The syntax is quite clunky on the Schneider…while the device has debugging symbols, you lack the ability to use C data structures by name. You have to do things the old-fashioned and painful way: make a socket structure in memory by malloc()’ing bytes, filling in the socket structure by hand, and then calling the C connect() function.

The upgraded firmware does remove the Telnet service. In my opinion, it’s not even worth upgrading the firmware yet though. While it does lower your risk ever-so-slightly, all that it really succeeds in doing is providing a false sense of security. An attacker that spends 20 minutes with the controller will learn that the firmware may be downgraded using the still-present FTP server and backdoors. The firmware image is simply a file that can be overwritten using FTP access. It also doesn’t fix the issues that our Metasploit module exploits, which just uses these two issues to grab user accounts for the system’s other services.

There are plenty of nice projects surrounding vxWorks now, including a vxWorks rootkit complete with a portscanner and other fun network penetrating tools. The HP rootkit was hand-coded in ARM assembler, but a port to PowerPC would be trivial. Schneider would be such an easy target because of their inclusion of debugging symbols with the firmware image. Very little reverse-engineering is needed to find the functions for changing a Quantum into a weapon.

I blame the WSJ reporter for the ridiculous story, “The director of the National Security Agency has warned that the hacking group Anonymous could have the ability within the next year or two to bring about a limited power outage through a cyberattack”. Either the reporter miscast what the NSA Director said or was dumb enough to bite on the story. Anonymous could do it now if they chose to as could anyone with moderate to strong hacking skills, desire, time and the willingness to get in a lot of trouble for causing a regional or larger blackout.

This week showed a good example of TippingPoint’s Zero Day Initiative (ZDI). Luigi Auriemma provided an ABB WebWare vulnerability to ZDI and got some money for it. The ZDI provided it to ABB who promptly developed a security patch. This week it was disclosed publicly.

More problems with pcAnywhere. Now pcAnywhere Nuke claims to crash a fully patched version of the product, and there may be more dangerous exploits coming. PcAnywhere is still widely used for ad hoc remote ICS access. A good reminder that every application and component on your ICS devices is part of the attack surface.

The fact that Congress has to deal with DCS and SCADA security for the critical infrastructure is another representation of failure by all in the ICS community, but in the US Government realm primarily by DHS as the responsible government agency.

Congress can’t be an expert in all fields and certainly not in something as arcane as control system security. It’s ridiculous that Congress and their staff have to try to determine how to solve this problem by crafting legislation. They should assign this and provide money to an agency who works the issue, and there is no evidence that the problem is DHS has failed due to lack of authority. To their credit, Congress sees little progress by the responsible agency in securing critical control systems and is trying to move the situation forward.

The best way to illustrate DHS’s failure is to look at the legislation itself, starting with the most damning evidence.

Prioritization and Focus – Sector Based Risk Assessment

Section 102 requires DHS to perform a sector by sector risk assessment “to determine which sectors pose the greatest immediate risk”. Sections 102 and 103 go into how this should be repeatable and ongoing.

Has this not be done yet by DHS? Shouldn’t DHS have handed the Committee the risk assessment report they have been doing repeatedly since they were founded? Shown how it has become more sophisticated and thorough over time. Show how it has driven the DHS programs. Show how they have measured success in terms of an improved security posture.

And I would hope they have a more than just an assessment as to what sectors should be prioritized. They should have:

a risk-based, tiered list of owner/operators in each sector (related to the crazy over reaction to a water pump in a small Illinois water utility)

a list of the key hardware and software technologies by sector, for example refineries use primarily Honeywell, Emerson and Yokogawa DCS. (related to the prioritization of ICS-CERT resources on key systems and applications rather than spending majority time with freeware HMI)

and possibly a list of the most important technical and administrative security controls missing in the top tier owner/operator systems

This prioritization requires making decisions such as a canal that provides the only water to a large, heavily populated region should receive a great deal of attention while a small, municipal owned water pump in Springfield, IL is handled by local authorities. It also requires the discipline to not jump on every vulnerability that can be tied to some control system function. Perhaps most of these should go through the normal US-CERT / CERT/CC process except for those in the key hardware and software list.

Stephan Beirer of GAI Netconsult briefs the S4 audience on the Smart Meter Gateway Protection Profile being developed in Germany. The effort was funded by the German Government and developed by utilities, vendors and consultants.

[vimeo http://vimeo.com/37150270]

(Note – last ten minutes are audio only)

For those new to the Common Criteria, Stephan provides some information on a Protection Profile – including the Security Functional Requirements and Security Assurance Requirements. He then discusses the key points in the Protection Profile. Some of the essential threats considered:

an attacker (local or remote) tries to gain access to the metering data or smart meter configuration/firmware

an attacker may try to intercept meter data or configuration/firmware during data transmission

an attacker may try to gain control of the gateway, meter or controllable local system

Project Basecamp highlights the fragility and insecurity in most PLC’s and provides tools so anyone can demonstrate and prove it. There should be no doubt that after ten years the ICS community needs to deal with this, but how?

This article finishes the series with what Government and Standards Organizations should do.

Government Organizations

I’m going to focus on the US Government, but much of this applies to governments around the world. In the US, the government is not responsible for securing the private critical infrastructure — at least not yet. So they cannot be directly blamed for the lack of progress over the last ten years.

The US Government does however have tremendous influence on the conversation and what C-level executives feel they need to pay attention. For the past ten years, INL and the other labs under contract to the US Government have performed security assessments of most of the major DCS and SCADA systems. They have known that the PLC’s and field devices were fragile and insecure. They were unsuccessful in making any progress in this area, as was everyone else in the community. An argument for keeping the problem quiet so the bad guys don’t know about it was reasonable, albeit a mistake with 20-20 hindsight.

This is now over. Stuxnet and Beresford were eye openers to anyone, including the bad guys, looking. Project Basecamp took it the next step by providing easy proof of concept tools to exploit PLC’s. There is no reason for the INL/DHS/USG to be reticent any more.

Which is why the latest ICS-CERT Alert is so unfortunate. It appropriately covers the risk of PLC attacks (Basecamp and others) and Shodan type searches. But nowhere does it actually address the root cause, fragile and insecure PLC’s and other field devices. When is the US Government going to come out and say these are a significant risk and that critical infrastructure Asset Owners should be working on a near term plan (1 to 3 years) to replace them?

As Chris Jager tweeted, perhaps an ICS-CERT Alert is not the place for a major policy change. Wherever they choose to announce this obvious conclusion, it is time. If the US Government, who is supposedly the expert in all things ICS security, refuses to state this as a necessary and urgent step, it is much easier for a C-level executive to continue inaction. They can point to there efforts to follow the ICS-CERT alerts and DHS guidance, and still ignore the PLC problem.

DHS and other government agencies are political arms with political skills. How about flexing those political muscles and get the vendors a bit of heat? When you think of all the Congressional hearings and Presidential edicts, it is always pointed at the Asset Owners. It’s not hard to generate heat at a Senate hearing:

Mr. Vendor, how is it possible that your state of the art PLC that you are selling today for use in power plants, pipelines and chemical plants, that costs 10x more than a laptop computer, doesn’t even have basic security features that prevent an attacker from shutting down the critical infrastructure by sending the simple message ‘turn off’? Why does my ATM card, home PC and smartphone have more security than your product? …

Project Basecamp highlights the fragility and insecurity in most PLC’s and provides tools so anyone can demonstrate and prove it. There should be no doubt that after ten years the ICS community needs to deal with this, but how?

Tomorrow the final part will cover what governments and standards organizations should do.

PLC Vendors

Eric Byres and others have written that it is not solely the vendors fault since they are in the business to sell product and make money. They will provide customers with what Asset Owners will pay for, and to date this has not been security. Our hope is that Project Basecamp will be a catalyst that will have large numbers of Asset Owners demanding a robust and secure PLC.

Security should actually be a boon for PLC vendors because Asset Owners will need to replace existing PLC’s much sooner than they normally would. Someday, hopefully soon, a smart VP in a PLC vendor will wake up and say we have a big upside opportunity here. Individual vendors cannot be individually blamed for the current situation because almost all have the same problem. What is required is an option for their customers to move forward if they care about process availability and integrity. The lack of a credible plan by most vendors, even three years out, is very depressing. Siemens has no announced plan, neither does Schneider or Rockwell. These three vendors alone have a massive installed base in the critical infrastructure.

So what should a vendor do?

Figure out if the current products have a future in a world that requires a robust and secure PLC.

If the software, hardware and architecture support upgrade, define the security controls that will be offered, when they will be available, what the cost is, and how the upgrade will occur. It may be a phased upgrade with tiers of security controls available over time. The Asset Owners can then decide when and if to upgrade.

If a new product is required, hello GE, define the security controls in the new product, when it will be available, and what the cost is. The security bar is higher if an Asset Owner is going to retire a PLC made obsolete by security early. I would highly encourage a threat model be developed to drive the security control design.

Document and implement your security development lifecycle (SDL). Customers and prospects will want to see the documentation and proof that is actually being followed.

Insure the SDL includes fuzz testing for communication stack robustness.

Consider submitting the product to ISASecure testing at Level 2. It’s by far the best PLC certification to date.

Project Basecamp tools was a big story, but we have covered that thoroughly on this site.

The other big stories, at least in the US, are happening in Washington DC. The Senate Cybersecurity Act of 2012 came was introduced by a bipartisan group of Senators. Homeland Security Television has a great, one-page at a glance summary. You can watch the Senate Testimony here. From CNN’s summary: “private companies that control such “critical infrastructures” would be identified the Department of Homeland Security and each individual company would be required to secure their own networks from cyberattack, and then “self-certify” in an effort to show the U.S. government it had complied. DHS would have the opportunity to spot check companies, and failure to secure could lead to civilian penalties.”

President Obama also released a proposed budget for next year that provided some clues. Patrick Coyle dug into the details and found on page 2118 of the budget rationale that ICS-CERT would increase from 9 to 12 full time employees in this budget. Other than small details like that, the budget related to ICS security remained about flat.

Dutch researchers showed how water control systems in the Netherlands could be hacked and maliciously operated to flood the Netherlands with water and wastewater. It sounded a lot like the Australian wastewater hack until you release how important water control is to the large areas that are below sea level. The impact of an ICS attack here would be huge.

Joel Langill joined the growing number of ICS security training options in announcing his new 5-day training course Understanding and Securing Industrial Control Systems. He also announced a 1-day introductory course. Joel is the former Infosec Institute teacher so it will be interesting to see who takes over that class. Now with a full field of private sector courses will DHS/INL finally admit they are competing with industry, which they are prohibited from doing, and stop their basic and intermediate training?

Travis Goodspeed has been wardriving around Knoxville, Tennessee looking for Zigbee, 802.15.4 access points with his mobile kit. He took some pictures that are worth a look. 802.15.4 is the basis of many DCS wireless systems including WirelessHART and ISA100, although those protocols do add security at layer 3.

Dale's Tweets

About Us

Digital Bond was founded in 1998 and performed our first control system security assessment in the year 2000. Over the last sixteen years we have helped many asset owners and vendors improve the security and reliability of their ICS, and our S4 events are an opportunity for technical experts and thought leaders to connect and move the ICS community forward.