For Secure & Robust ICS

Goal: Help Owner/Operators select the best anomaly detection solution for their ICS.

It sounds simple, but after getting numerous demos and pitches from vendors, the almost unanimous contention from each vendor was that their solution was the best. Why? Because they go deeper, understand the protocol, system or user better than the competition, who were often denigrated as smoke and mirrors. Most would follow up with a specific example of their detection capability, but none provided even the start of a way of comparing solutions short of installing multiple solutions and testing.

I describe the technical marketing to date in this space as emphatic assertion. Ours is better. No really ours is better.

At S4x17 we had two sessions on “How Deep is Your ICS Deep Packet Inspection” with the goal of getting to a consensus approach or at least examples of how to perform a product comparison. While the sessions gave good anecdotal examples, we were no closer to a methodology.

In two weeks, I’ll be at our S4xEurope event (June 1-2 in Vienna) and try again. There is a promising session from Sentryo on Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. It uses the automobile CAN network, and falls into the great examples category. I also will be moderating a panel with technical representatives from Claroty, Nozomi and Security Matters. While it’s not a hostile interview, I’ve warned the panelists that I’m going to push them on this evaluation issue and call BS if necessary. (Note: I welcome any suggested questions for the panel, and we video the event so you will get to see it late June on our S4 Events YouTube Channel)

As preparation for the panel I wanted to have a list of vendors offering what I’m calling an ICS Anomaly Detection solution and created a LinkedIn Post trying fill out my original list of 14. That list, see below, grew to 22 even with removing suggestions with the three following characteristics:

Standard IDS/IPS Solutions – We are believers in IDS/IPS signatures, after all Digital Bond wrote the first basic ICS signatures and they are still widely used. Classic IDS/IPS solutions are not included because this new product category is focused on learning or knowing “normal” network, device, application or user activity and identifying variations from this norm that could be indicative of an attack.

Perimeter Security Solutions – There are a number of ICS gateway solutions with some impressive ICS protocol intelligence or effective one-way technology. Perimeter security products are essential to an ICS security program, but not in this new anomaly detection category.

Primarily IT Security Solutions – Most of the mainstream IT security products are adding some ICS intelligence, and at some point they could be competitive with ICS focused products.

All three of these are judgment calls, and I welcome any comments where you think I’ve missed the mark.

As the LinkedIn article comments came in I decided to add two columns to the list:

Country of Origin … this is interesting to identify where the startups are coming from and also becomes important with the increasing cyber nationalism

Funding … you have 20+ vendors competing for a very new, and some would say unproven, market. Having enough runway to survive until the market grows will be key, although burn rate is just as important. And yes, there will be carnage.

So here is the list, and I expect it will require updating this week as more companies and better information comes in.

It’s hard to miss that a large number of the companies are Israeli (9 of 22), and that most have raised money in the last 12 months. However what I want to focus on is the difficult situation facing any ICS Owner/Operator who is considering buying an ICS anomaly detection solution, especially when they all are saying close to the same thing about why they are the best.

It’s not getting better, and the number of vendors offering ICS anomaly detection solutions continues to grow in numbers and angel/venture funding.

How is an asset owner to determine what anomaly detection approach, if any, is right for them?

The first decision points are simple:

Are you ready for ICS anomaly detection?
If your ICS security protection program is not mature and under control, then you’re not ready. If you are not doing basic detection, such as monitoring firewall logs and endpoint protection, you are not ready. If you don’t have the detection and incident response team to assign to anomaly detection, you are not ready.

Does the ICS anomaly detection support your deployed products and protocols?
All of the vendors clearly state what they support, but some are a bit vague on when the support will be available. The protocol work is fast and furious.

After those three questions the evaluation runs into significant difficultly. I have had numerous demos, conference calls and discussions with ICS anomaly detection vendors, but I must say the arguments the vendors give as to why their solution is better is typically emphatic assertion (we support and alert on more of the protocol than the competition) and identical to what I hear from their competitors.

We tried to make progress on evaluation methodologies with two “How Deep Is Your DPI” sessions at S4x17 on Stage 2. The sessions gave good and specific examples on how ICS anomaly detection can detect cyber attacks and incidents, but really didn’t move the evaluation challenge forward much.

First, I’ll be interviewing a panel of technical vendors on stage including Damiano Bolzoni from Security Matters, Andrea Carcano from Nozomi Networks, and a third panelist to be announced. I’m working on my pointed questions and followups in an attempt to get past the generalities, and welcome any suggestions. My focus is going to be on the evaluation criteria, and how they are using machine learning or other techniques to identify potential cyber incidents.

Second, we have a very promising session from Jean-Cristophe Testud of Sentryo Security entitled Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. Much of the work today in ICS anomaly detection is related first to communication pairs and patterns, and second to identifying high impact requests (something we did poc with the DHS funded ICS signatures in 2006).

Since this is a vendor session, we required early submission of the presentation to check for commercialism and content. It’s great. It shows modeling/learning of automobiles via CAN traffic, and detecting false data and commands. This session shows the power of the structured machine learning and also shows how a vendor could potentially provide a listing of capabilities per protocol in this area.

We have learned in recent years to leave a slot or two for late breaking attacks on ICS or hot research in the S4 agenda. Ukraine has helped fill this spot now for the second year in a row. We know that something happened again to the Ukrainian Power Grid, and there is still much that is not known or not yet public as researchers and analysts are once again working hard over the holidays.

So who is best to put on stage to reveal and discuss the latest information and analysis?

Answer: The people closest to the information and problem/challenge.

So we will have Marina Krotofil, who hails from Ukraine and now is working for Honeywell, on stage and
Oleksiy Yasinskiy from ISSP in Ukraine on a live video feed. Marina and Oleksiy may choose to add additional people on stage or via video from Ukraine.

I’d like to be able to give you more of a description or feel to what you will learn, but it likely would be out of date in the next day or two. What I can say is you will get the latest and most detailed information known on January 10th.

2016 was a turning point with secure ICS protocols. For a while it was limited primarily to OPC UA and DNP3 SA, but 2016 brought us a secure version of CIP / Ethernet/IP, Secure Modbus and a couple of others that will soon be unveiled. This should be enough critical mass to force the other protocol bodies to do the same in 2017 – 2018.

Schneider Electric has developed a Secure Modbus protocol that they are proposing to the Modbus organization. It will support authentication and encryption of course, and Daniel will explain how. What I found most interesting is the use of certificates to enforce roles at the PLC/RTU itself. This delves into a PKI which can be a morass. So I’m looking forward to hearing how this will be implemented and managed.

Secure SCADA Protocol for the 21st Century (SSP21) with Adam Crain and Rich Corrigan

After beating on DNP3 and other protocol applications as part of Project Robus, Adam decided to work with Rich to come up with a more secure protocol. SSP21 is intended to fill a technology gap where existing technologies like TLS are not applicable, namely for serial communication channels and endpoints with limited bandwidth and/or processing capabilities

Second – My thoughts on who should consider participating in the S4 ICS CTF.

A person with hacking skills, but little experience in ICS. The flags will give you guidance on what an attacker would actually try to do once they can get to an ICS.

A person responsible for defending an ICS. Even if you just spend time understanding the flags you will learn many of the end goals and techniques that will be used against your ICS if an attacker can gain access to it.

A person with great ICS hacking skills. You will find this a challenge and perhaps you can win the S4 Black Badge.

Third – Some tips from Reid for CTF participants:

A successful team will need a variety of skills, including the ability to analyze industrial controls, to basic network scanning, to lockpicking, as well as solving more traditional CTF problems.

Some challenges are purely control systems focused, such as identifying configuration items in controllers or analyzing oddities in ICS protocols. Some of these control systems challenges will have a cyberphysical element — as teams solve the problems, they may want to watch process control equipment to see how their finding helped attack a process. A few of these will involve ICS Foreverday vulnerabilities.

Other challenges involve incident response: analyzing traffic from compromised systems. Bring your traffic analysis hats for these. We even have RF analysis flags. We will have a handful of SDR receivers and will provide hints for how to search for these flags; players want to familiarize themselves with the RTL-SDR prior to coming.

There are two sessions at S4x17, Jan 10-12 in Miami South Beach, covering actual ransomware incidents in ICS. Marcelo Branquinho of TI Safe will go over two case studies that occurred in South America on the Main Stage, and RSA will discuss an ICS ransomware case in the US that also involved the FBI. All three cases will be anonymized, but there is some very interesting detail on how the companies dealt with the incidents.

This article comes on the heals of a ransomware incident on San Francisco’s Muni Train and Bus ticketing system, and likely a large number of other ransomware attacks that are never made public. I don’t think it is a bold prediction that ransomware in ICS will increase.

Given that change is minimal in ICS, even a quarterly high confidence, off network backup is likely to be sufficient for recovery without unacceptable loss of information. High confidence and off network are key. We often find in assessments that the hot standby system is used as the “backup”, and interview and inspection shows more of an occasional good effort backup spread over servers, laptops and USB drives.

The bigger issue with ransomware in an ICS may be around the time to recover and the confidence in the ability to recover. Is the Recovery Time Objective (RTO) truly an acceptable outage time and is the asset owner certain it can be met? This also has ramifications for the attacker. They will need to shorten the time they give for payment, which means the asset owner will have a shorter time to decide to pay or not … another good scenario for a tabletop incident response exercise.

Should be two interesting sessions and lots of good discussion at S4x17.

A South Florida High School Class will go through two days of hands on automation and security training with Matthew Luallen and the CybatiWorks kit, and then 12 of the students and their teacher will come to the Main Stage on Thursday to discuss the experience. They will hang around at lunchtime if you want to meet and talk to them.

Matthew Luallen deserves big thanks for first putting together the CybatiWorks kit and program, second working with the school we connected him with in Palm Beach, and third volunteering his time to perform the training. Digital Bond is purchasing six of the CybatiWorks kits for the course and will donate them to the high school after S4.

Our hope is that some of the S4x17 audience will be inspired by this and look for ways to improve on this effort and potentially develop something that is scalable. One of the larger challenges is if something like this is successful, meaning students learn from it and get excited about doing more of this type of learning and work, how do you make something like this available to 1,000 students? 1,000 students in each state? It’s more than just raising the money to buy the kits. It’s training and supporting the teachers. Developing courseware and likely a host of other items.

Marc Blackmer of Cisco, a Cabana Session Sponsor, is giving another important and related talk on the Sponsor Stage entitled: Mentoring for Fun and Non-Profit. He will talk about his experience in creating 1NTERRUPT, a free, non-profit cyber security program for students ages 14-22 that emphasizes creativity, community, and meritocracy. 1NTERRUPT is based in Worcester, MA, and now has chapters in San Francisco, Atlanta, and Portland.

OSIsoft is back again as a S4xCTF sponsor, and they are bringing back Killer Robots, Inc. with new and unsolved flags from last year. Enter Harry Paul of OSIsoft to give you some information and hints to help you get some of the PI System related flags in the S4x17 CTF.

The S4x17 Killer Robots CTF environment is designed to be an interactive, fun source of industrial security challenges. After all, CTF is a great way to explore and defeat ‘forever’ day configuration issues. This year the OSIsoft team has improved and expanded the PI System environment, planting flags inspired by case studies, new security features and threat models.

Below we have a summary of the PI challenges from last year. OSIsoft provided 11 of the 43 total flags for the competition. There were 5 flags left standing at the end of the competition and 4 flags that were only solved by one team. The most successful competitor captured 450 of the possible 2025 points from the PI challenges.

Flag

1

2

3

4

5

6

7

8

9

10

11

Points

25

50

100

100

125

50

125

300

300

500

400

Successes

16

6

1

1

1

1

0

0

0

0

0

Reviewing the logs in our environment revealed that many teams did perform reconnaissance, but did not progress. Perhaps the low success rate of the competitors has gone to our heads, so this year we are upping the ante. The first (if any) team that captures the mysterious, illustrious “Golden PI” flag, will win the opportunity to deliver ~3.14 pies to the faces of the OSIsoft security advisory team in attendance. You heard right, this is your opportunity to exact sweet revenge on a vendor!

Want to learn more? Every Wednesday in December we’ll give an inside look at the CTF environment on the PI Square Security Forum, providing background and perhaps even a few hints along the way. Search for the S4x17 tag to get all posts related to the event.

I am pleased to announce that Justine Bone of MedSec agreed to an interview on the Main Stage at S4x17. Vulnerability disclosure is and has been a contentious topic in ICS. I generally don’t write much about it because the person or organization that finds the vulnerability decides what is the responsible and appropriate disclosure. Full stop.

We have seen all sorts of disclosure approaches at S4, and even had a bit of a controversy ourselves around pointing out insecure by design issues in PLC’s and RTU’s as part of Project Basecamp at S4x12. However this or any other type of disclosure has not been as aggressive and controversial as the MedSec/Muddy Waters disclosures of vulnerabilities in St. Jude Medical’s devices.

In order to help address patient safety, we have chosen to depart from standard cyber security operating procedures in order to bring this to the public’s attention and to ensure that St Jude Medical responds appropriately and with urgency. We have shared our research with an investment firm, Muddy Waters Capital, that is helping us deliver this message.

The time has come for us to re-think the way cyber security is managed. We acknowledge that our departure from traditional cyber security practices will draw criticism, but we believe this is the only way to spur St Jude Medical into action. Most importantly, we believe that both potential and existing patients have a right to know about their risks. Consumers need to start demanding transparency from these device manufacturers, especially as it applies to the quality and functionality of their products.

Muddy Waters publicly shorted the stock and issued analysis saying they expected revenue to decrease up to 50% over the next two years due to recalls and remediation costs, and “MedSec is receiving compensation related to investment profits from the funds Muddy Waters manages”.

There are a lot of questions around this approach in terms of legality, ethics, disclosing vulnerabilities without detail, effectiveness in getting the issues fixed, impact on the security research community and much more. I will have no difficulty coming up with questions to fill the 30 minute interview, but we decided to open this up to the ICS security community. What would you like to see Justine Bone asked in the onstage interview?

The industrial firewall and ICS anomaly detection markets are getting very crowded. The industrial firewall market is older, but it is still expanding both in specialized ICS firewalls and enterprise firewalls adding ICS protocol support. The ICS anomaly detection market has exploded with a new entrant almost every month and millions of dollars of funding.

The benefits of these product categories are heavily based on their ability to perform deep packet inspection (DPI) of ICS protocols. Firewalls do this for more granular control of a security perimeter (and some IDS/IPS), and anomaly detection rely on DPI to identify unusual or potentially damaging use of ICS protocols.

These products are typically promoted by the breadth and depth of the ICS protocol support. The breadth is easy to compare and somewhat useless. A vendor can easily list the protocols they support at some unspecified level of depth. I say breadth is somewhat useless because an ICS asset owner doesn’t care if the vendor supports 10 or 30 protocols; the ICS asset owner only cares if the product supports the protocols they use.

Depth in DPI of the protocols an asset owner uses should be one of the key decision factors, along with company viability, ease of use, reporting, support, interoperability with SIEM’s, … Depth can vary figuratively from inches to a mile deep, and depth can vary a lot per protocol in the same product. We worked with one client considering an enterprise firewall with tremendous breadth of ICS protocol support. The firewall vendor was only checking the TCP port number and a single byte in the request packet, inches deep, in the protocol our client was most concerned with. We know that the same vendor has very deep DPI for other ICS protocols including proprietary extensions of the protocol to cover engineering work station actions.

Talk to the anomaly detection vendors and they will typically tell you not only how completely they inspect the ICS protocol, but also how they do this to a much greater degree than their competitors. When asked for more details and reasoning it devolves into emphatic assertion, and they cannot all be right. It is likely that simple protocols have similar levels of depth, but more complex protocols will vary as will support for proprietary extensions.

At S4x17 we are trying to help asset owners and the ICS community compare and contrast ICS DPI with two sessions on Stage 2 titled How Deep Is Your ICS DPI? The speakers have been challenged with developing a structured method to evaluate the depth and value of the DPI of an ICS protocol. Ideally this would come down to a method of comparatively score the solutions. Given the number of vendors and asset owners looking at this issue we are hopeful we can at least narrow down the approach to comparisons.

Dale's Tweets

About Us

Digital Bond was founded in 1998 and performed our first control system security assessment in the year 2000. Over the last sixteen years we have helped many asset owners and vendors improve the security and reliability of their ICS, and our S4 events are an opportunity for technical experts and thought leaders to connect and move the ICS community forward.