How to Manage the Ex-Employee Insider Threat

A developer or operator leaving your company is always a harrowing event. More than likely they had access to your production environment, so you engage your standardized process for revoking their access. But how can you be sure everything is truly cleaned up, regardless of whether you suspect they would be malicious or not?

Automated systems like Chef can be very helpful in disabling login access, but typically miss things like scheduled jobs (cron) or long running abandoned login sessions (it’s always interesting when you find a forgotten detached tmux session that’s been running for three months). These types of rot are easy to miss in deployments of hundreds of servers, let alone thousands or tens of thousands of servers, where there is no possible way for operators to have an intimate relationship with individual servers.

The solution requires two approaches: actively monitoring for ex-employee activity in your environment (IDS) and hunting through retained workload information for traces of activity (audit).

Active monitoring for ex-employee activity

You already should have a list of users who need their access revoked, ideally from your internal ticket or work tracking system. This list is crafted easily as a Threat Stack alert rule, which you can apply to your base Rule Set. In this example, we will apply our ex-employee alert rule to the Default Policy.

This screen shot shows us configuring the initial New Alert Rule for our policy in the Rulesets UI. The Alert Title has a clear indication of what is happening, including the username that was observed in the system. The Alert Description provides suggestions for your Incident Responder to follow, regardless of whether that person is a general operator or dedicated security staff. We have also set the priority as high as it will go, making sure that we will get notifications for the event via PagerDuty and e-mail.

The Alert Filter itself is pretty straight forward. We just list all of the system usernames that should no longer be active in our environment. We could apply additional filters, but we would rather catch all of their activity in the environment since there should be none.

Clicking on “More Alert Options” brings us to this next screen. The only modification that we need to make is setting Aggregate Fields to user. This means that we will get separate alerts for each ex-employee activity observed, allowing us to prioritize our response. For example, if our team is bandwidth constrained we might prioritize an ex-employee who had access to customer data versus a lower level operator who did not.
Now when we view our alerts we can see actionable, high priority activity that we must remediate.

This will tell us about any future ex-employee activity in our environment. But what about past activity?

Hunting for traces of ex-employee activity

There are fundamentally two different types of activity that we are hunting for at this stage.

In the window between the employee leaving the company and us creating the above Alert Rule, what actions did they perform? This is especially tricky if you had to terminate the employee, as they may have suspected an impending employment action.

Prior to the employee leaving the company, what actions did they perform in the environment? These actions may range from unknown technical debt (old maintenance scripts they left behind) to malicious behavior (data theft, unauthorized workloads, etc.).

The first type of activity is a much more straight forward one as the time window should be fairly small, potentially minutes or hours long. If your standardized process for removing an ex-employee’s access lasts longer than this then you need to rework that process and find better tools.

For example, we might do a search for user=”elliotalderson” && timestamp > 1438104640000, which would show us all events for the user since 1:30pm Eastern on 7/28/15 (timestamps are in milliseconds). We can then select the Executables button at the top of the screen, which will show us a list of executables in order of frequency.

While the most frequent executables can be interesting to suggest what the user typically does in the environment, it’s often the least frequent executables that tell you the most – especially in that window between them leaving and their access being terminated, when they might try to export data or leave behind files.

Another useful search that applies to both windows of activity is network events that reach out to the Internet. For example, let’s say that we are concerned that our elliotmurphy user might have stolen data and uploaded it to an outside server. We could start with a search for user=”elliotmurphy” && type=”connect” which would show us all of the connections that the ex-employee made. Then, leveraging the Executables and IP Addresses statistics at the top of the page, we can begin to filter the results.

Executables such as ssh, scp, nc, fsync, and curl should take top priority as they can transport data.

IP addresses outside of our internal blocks should also take top priority. Do not just look at WAN addresses though, as the user might be staging files in publicly available servers so that they can pull them down at a later date. An example search might be type=”connect” && command=”scp” && ip=”10.10.10.10” if that is our web server’s IP address.

Also give VPN IP addresses a high priority. For example, if all VPN clients are placed in the 172.16.0.0/16 private block, then a search for ip starts_with “172.16.” will show you all inbound and outbound connections. Filter the results carefully with the statistics at the top, as files could be exfiltrated by either a push mechanism, which would have user=”elliotalderson”, or a pull mechanism, which won’t necessarily have the username.

Conclusion

If the notion of the “ex-employee insider threat” wasn’t high on your list of concerns, reading this account of the Hacking Team breach and data loss should change that. Addressing this specific threat is an example of the deep level of visibility a solution must provide.

As the Chief Technology Officer at Threat Stack, Sam is responsible for leading the Company's strategic technology roadmap for its continuous security monitoring service, purpose-built for cloud environments. Sam brings highly-relevant experience in distributed systems in public, private, and hybrid cloud environments, as well as proven success scaling SaaS startups. Sam was most recently the CXO at Cloudant (acquired by IBM in Feb. 2014), a leader in the Database-as-a-Service space, where he played a senior technical and product role.