The US government’s ICS-Computer Emergency Response Team (ICS-CERT) tracks and publishes Security Advisories for known security vulnerabilities found in industrial products. In the entire decade prior to the discovery of Stuxnet (July 2010), ICS-CERT published 5 security advisories involving 3 vendors.

Compare this to 2011, when there were:

• 215 publicly disclosed vulnerabilities

• 104 security advisories

• 39 vendors involved in those security advisories

By late 2012, the total publically disclosed vulnerabilities topped 569.

And remember that these vulnerabilities are typically disclosed to the world prior to ICS vendors having patches available.

Furthermore, 40% of disclosed vulnerabilities include working attack code. This means that individuals can download exploit tools and run them against a target with little understanding of control systems or the consequences of their actions. And download and attack they do - ICS-CERT reported over 20,000 reports of unauthorized internet access to control systems in the last half of 2012.

Welcome to the Industrial Security Patching Treadmill!

Since security researchers, hackers, and issues have essentially migrated from the IT environment, it’s not surprising that we look to that world for a solution. There, security vulnerabilities are addressed by applying software patches - a constant cycle involving multiple patches over the life of a product. In fact, the typical IT computer needs patching (with a full reboot) at least once per week.

You don’t need to be a SCADA/ICS expert to realize that shutdowns of that frequency just aren’t feasible for critical infrastructure control systems.

So How Many Patches Does a Control System Need?

One might argue that a control system requires fewer patches than an IT system…or that the software footprint is smaller, the code quality better… If so, then maybe the patching cycle could be synchronized with annual maintenance shutdowns. In this scenario, patching could be a workable solution to address software vulnerabilities.

To determine if this was an option, in 2008 I participated in the analysis of a U.S. refinery process control network (PCN). There were 85 computers on the refinery PCN, and a similar number of industrial controllers. Although we could only gather reliable data for 78 of the computers, we determined that they were running 272 distinct processes/applications.

A search of the National Vulnerability Database (NVD) found that 48 of these processes had one or more serious security vulnerabilities. Across the refinery PCN, there were 5,455 publically known vulnerabilities, an average of 70 per machine. An aggressive Windows OS patch program reduced this number by almost 50%, but that still left 2,284 published vulnerabilities remaining. Why? Because the applications involved did not have a means of automated patching.

Latent PLC Vulnerabilities Are Not All Disclosed at Once

And let’s not forget the latent vulnerabilities lurking in the control system. Academic research tells us that most commercial software contains 3 - 10 defects for every thousand lines of code (KLOC), and that 1% to 5% of these result in vulnerabilities. That works out to between 0.03 and 0.5 vulnerabilities per KLOC.

So what does that mean in real life?

Take Windows XP, for example…it contains about 40 million lines of code (40,000 KLOC). As of October 2012, about 1106 moderate or severe vulnerabilities have been listed in the NVD for Windows XP - that’s a Vulnerability/KLOC ratio of about 0.0276.

Whatever your experience using Windows XP, seems it’s on the low end of vulnerabilities and pretty good from a security point of view!

Looking back at the history of Windows XP vulnerabilities, we have to assume that SCADA/ICS vulnerabilities won’t all be disclosed at one time. We’ll likely see a relatively small number of disclosures in the first few years, as researchers begin to investigate the products in the industrial space. Then, after SCADA/ICS products have been exposed to widespread security scrutiny, a virtual avalanche of vulnerabilities may occur, resulting in the need to install control system patches on a weekly basis.

We Can’t Ignore SCADA/ICS Firmware

It’s also obvious that the firmware in PLC and DCS controllers will also have vulnerabilities and will require patching. Controllers typically contain between 1,000 KLOC and 5,000 KLOC of firmware. Based on the analysis used above, this means that each is likely to contain between 30 and 150 vulnerabilities. If the vulnerability disclosure curves are similar to those we’ve seen in the IT sector, we can expect a low number of patches in the immediate future, followed by an epidemic in a few years.

The above analysis clearly indicates that the frequency of patching needed to address future SCADA/ICS vulnerabilities in both controllers and computers is likely to exceed the tolerance of most SCADA/ICS operators for system shutdowns.

Tune in to next week’s blog and learn about the impact of patches, what happens when there are no patches, and why many SCADA/ICS customers simply don’t want to patch...

Do you think that patching is a workable solution for securing SCADA/ICS control systems? Do you have any patching success or horror stories to share? Let me know your thoughts.