From Agent Install to Mean Time to Know in Less Than 5 Minutes

They say timing is everything — and when you go from installing an agent to detecting and remediating a security breach in less than 5 minutes, it just doesn’t get any better.

We want to share an awesome story about how one of our customers recently caught a breach within seconds of installing Threat Stack.

Summary

Here’s a summary of the main steps we took and the times it took to accomplish them:

Install Threat Stack Agent & View Sev 1 Alerts 20 seconds

Confirm Threat & Review TTY Timeline30 seconds

Determine Cyber Kill Chain Steps & Remediate60 seconds

1: First View, Severity 1 Alerts: 20 Seconds

Within 20 seconds of installing the Threat Stack agent, we observed the following alert sequence:

We knew immediately that the organization had been breached because of the numerous Severity 1 alerts, and because of the types of events that were generating these:

Abnormal login failures

Kernel modules installed

www service running shell processes

Outbound communications to known bad IPs

2: Confirmation & Review of TTY Timeline: 30 Seconds

After viewing the alerts, we started a closer investigation, beginning with the Kernel Modules Installed alert. To do this, we clicked on the alert to view the event details, and then clicked on the Process Details to look at the TTY timeline (shown below) to quickly see what the exploit had done pre and post in that session.

From here we noticed that this process was replacing the binaries with trojan system binaries. Since these binaries are monitored by Threat Stack’s File Integrity Monitoring functionality, we should have received file OPEN and MODIFY alerts as well. We quickly confirmed that we had received these as Severity 3 alerts as shown below:

On a close examination of the files that had been changed and the timestamps of those changes, one configuration file change — DbSecuritySpt — caught our eye. (See the next screenshot.) The creation of an init.d service by a process called breeb was certainly suspicious and was quickly confirmed to not be a service or tool that is expected or known to be running on this, or any servers. Further investigation revealed that DbSecuritySpt is similar to or a variant of a known botnet, Linux.BackDoor.Gates.5.

3: Determining the Cyber Kill Chain & Remediation: 60 Seconds

A quick Google search matched what we saw closely enough so we could determine the cyber kill chain steps:

Login Failures indicates a wide-open SSH port, which was the entry point of the attack.

Apache running shell indicates a web shell exploit.

Kernel module indicates a payload trying to hide its tracks.

External Communication is command control.

The remediation, of course, was to close the wide-open security group.

The Importance of Minimizing Mean Time To Know

Mean Time To Know (MTTK) is extremely important in security operations: the shorter it is, the more an attack’s impact can be reduced.

We have all heard stories and seen statistics about infection or compromise dwell times. These are usually a number of months rather than minutes or hours. The longer the dwell time — or in other words, the longer the MTTK — the more likely it is that the damage will grow exponentially.

To give some insight into the seriousness of dwell time, our server Details Page showed the following connections, which indicated that a large number of communications had taken place within just a few minutes of the breach:

But there’s more. . . We also noticed that an open source rootkit detection tool had been installed on the host but had not detected the attack. This probably occurred because the attack that we identified through a quick Google search was sufficiently different and was therefore likely a variant on the actual attack, meaning that the existing signatures would not have picked up this attack for days, if not weeks or months. With Threat Stack installed, the MTTK was less than 5 minutes.

15 years of experience as a security engineer at Trusteer (IBM Security), Core Security Technologies, and Threat Stack, has made Anthony a valuable member of Threat Stack's growing Oversight team. Anthony helps our customers deploy, configure, fine-tune, and manage their continuous security monitoring so they can run secure and compliant, without sacrificing time and resources.