Industrial control systems (ICS), much to the industry’s chagrin, are not immune to data breaches, ransomware attacks, viruses, malware, insider attacks or any other form of assault. On top of that, there are some experts that say they need tangible proof attacks really happen. Well, there is.

Just take a look at the Verizon 2016 Data Breach Investigations Report (DBIR). The report details 16 common breach scenarios and the cases are each told from the perspective of the stakeholders involved, such as corporate communications, legal counsel, or the human resources professional.

In one poignant scenario, a manufacturer fell victim and had to get their house in order fairly quickly. Here is what the report said about this one incident:

“A company, we’ll call Gator-Grasp Fasteners, retained the Verizon RISK Team to perform a health check of their industrial environment. This particular customer was in the business of fabricating specialized fasteners, which were required to pass very specific engineering requirements, such as meeting or surpassing certain strength, tensile stress, mechanical properties and material content thresholds.

“At the onset of the health check, Gator-Grasp Fasteners’ automation engineers expressed skepticism and mild dissent, arguing that a “health check” was not necessary. In their many years of being on the job, the “patient” had always functioned well and had shown no signs of being “unhealthy.” So why mess with things? They assured their management that the Operational Technology (OT) environment was secure and that they expected there would be no significant findings. After all, the automation engineers were experts and they knew what they were doing. Nonetheless, management insisted and the automation engineers reluctantly agreed to work with the RISK Team.

Not in my house

“As with any engagement, there was a kick-off meeting, which was used to introduce everyone, set initial expectations, discuss the in-scope environment, request additional information and schedule the onsite visit.

“The requested information included a list of network segments, IP address ranges, IP address assignments, and an asset inventory. “The Gator-Grasp Fasteners Team was instructed not to create any new documentation in order to avoid a situation where the creation of new documentation would potentially mask a procedural deficiency. In assembling the requested documentation, Gator-Grasp Fasteners quickly realized that what it did have was inadequate.

“During the on-site visit, the automation engineers, the RISK Team’s critical infrastructure protection/cybersecurity (CIP/CS) specialists and other subject matter experts (SMEs) discussed the various OT systems, in-place security measures and other operational procedures. This included processes and practices (aka “institutional knowledge”) that are followed, but were not necessarily documented. These discussions revealed that over the past few months, the network seemed “sluggish,” which the automation engineers and SMEs attributed to older, legacy equipment. With an understanding of the situation in mind, we visited various locations where we walked the manufacturing floor and made additional observations.

“One of the first things we noticed was some OT systems had anti-virus protection while others didn’t. For those that didn’t, we were told that, since they were isolated, they didn’t need protection. Incredibly, when we looked at the anti-virus logs on the OT systems that had malware protection, we found them replete with malware detections, deletions, and quarantine alerts. Of the 57 systems in total, 33 systems had at least one malware alert, and many had multiple alerts.

“When we inquired about these alerts, we found that the automation engineers and operators were well aware. They reasoned that since the malware protection was correcting and “repairing” the problems, everything was acceptable. We explained that there was clearly an underlying problem leading to the repeat infections and recommended a more detailed review to identify the root cause.”

Need logging

“Gator-Grasp Fasteners had no documented Incident Response (IR) process for investigating incidents, so we took the lead. The company did not have a centralized logging solution and what devices did log did not provide insight into how the malware was getting into the network. The problem? We needed more visibility.

“With the cooperation of Gator-Grasp Fasteners, we set up a Switched Port Analyzer (SPAN) port and deployed a passive network analyzer to collect and analyze the traffic. Using indicators related to the identified malware, we reviewed network traffic and quickly identified multiple potentially infected systems. As we expected, the network traffic revealed malware infections associated with the legacy OT systems that did not have anti-virus protection. Further analysis revealed that a number of misconfigurations existed – which had allowed unauthorized network communication.

“The infected systems, many of which were very actively searching for new systems, were a good candidate for the “slow network” problems identified during earlier interviews. Using the collected network traffic, we ran statistics on data transfer rates and quickly realized that the scanning attempts were saturating legacy network connections with probes. With a concrete list of infected systems, we targeted the population of compromised endpoints.

“Despite the widespread infection, Gator-Grasp Fasteners had been fortunate. Review of the malware resident on each system revealed common drive-by infections, all targeted at stealing banking credentials. As none of the infected OT systems were utilized for anything other than process management, it was unlikely that further damage had occurred. The network trouble was an unintended side effect of the malware’s attempts to find new systems compounded with overly permissive firewall rules.

“We provided a list of known infected systems to Gator-Grasp Fasteners, which quickly began rebuilding them from known good images. To keep remediated systems remediated during this process, we continued network traffic monitoring for known indicators and behaviors associated with the identified malware. With the current issue well on the path to being resolved, we turned our attention to the uninfected, but still “troubled” OT systems.”

Verizon RISK team broke down some recommendations into three categories covering the company’s entire OT environment:

Unnecessary legacy systems in unmanned locations. These systems ended up removed from the network and decommissioned. These were difficult to track down as they were not documented, making them hard to find, which ultimately delayed the containment and eradication activities.

Necessary legacy systems unable to end up protected by an anti-virus solution. The RISK team manually removed the existing malware and the systems ended up hardened from a best practices standpoint. Stringent firewall rules ended up deployed to prevent access to and from these systems, designed to limit the reach of any future compromises.

New systems not patched or protected by an anti-virus solution. These computer systems ended up patched and malware protection installed.

Best practices

Just as in the non-ICS world, a security incident can cause damage to brand reputation, loss of competitive advantage, legal or regulatory non-compliance issues, considerable financial damage, and harm to the environment and community.

The biggest lessons learned could be summed up in one automation engineer’s comments: “…Well, being here for over 25 years, I thought I knew all the ins and outs. I didn’t consider documentation very important, but in the end, I realized, there was a lot that I didn’t know, and what I didn’t know ended up being a big part of the problem.”

The RISK team found there were multiple corrective actions that Gator-Grasp Fasteners needed to take to shore up their detection, mitigation, and response efforts, including:

Perform IR planning. An IR Plan is critical to resolving security issues by providing direction and guidance to responders.

Conduct first responder training. Train those most likely to identify security issues about the IR Plan; educate them to collect information and triage immediately.

Harden OT systems. Devices with overly permissive default configurations should end up reviewed and unneeded configuration options disabled, to reduce the risk of misuse.

Patch and patch often. Develop a patch management program to properly secure assets and networks. Security patches fix known vulnerabilities and mitigate the spread of malware.

Utilize anti-virus/intrusion detection system (IDS) protection. Install a host-based anti-virus solution or intrusion detection system on all IT/OT systems and keep the definitions up-to-date.

Configure logging, monitoring and alerting. Centralize logging from all devices into a single location and periodically review logs for signs of suspicious activity such as anti-virus alerts, failed log-in attempts, or network communications involving external systems.

Maintain IR/disaster recovery plans. It is essential to have well-documented and run-tested IR and disaster recovery (DR) Plans. If not, the response and recovery process will be disorganized, potentially incomplete, and take much longer.

Gregory Hale is the editor and founder of Industrial Safety and Security Source (ISSSource.com), a news and information Website covering safety and security issues in the manufacturing automation sector. This content originally appeared on ISSSource.com. ISSSource is a CFE Media content partner. Edited by Joy Chang, CFE Media, jchang@cfemedia.com.