Identifying Insider Threats Within Your Docker Containers

Docker. It’s a thing. A big thing. Actually, it’s a bunch of little things. Things called containers that like to pretend they’re running in isolation. Except they’re not. Nevertheless, they’re still hot right now.

Threat Stack announced the release of its Docker integration during Amazon’s Re:Invent conference of 2015. This feature augments detected host events with Docker information when the Threat Stack agent identifies the event as originating from a container. Augmented information consists of the Docker container ID and the image name. We collect that data with a host-based agent that does not stick some additional agent into each container. Per-container agents would cause performance issues for typically small footprint containers. Our daemon runs in user space and does not hook into the kernel, allowing us to stay lean and lightweight. Please allow me to explain a bit about how this all works.

Docker provides a great remote API. Its daemon listens to unix:///var/run/docker.sock by default and the API tends to be REST. This means we can communicate with and inspect our containers by sending HTTP requests over the local socket. Consider that I have a container with an abbreviated ID of 034fe81c01e1. I can obtain information about the container by executing the json query:

That returns more JSON, but this time it lists the processes executing in the container. Something may seem a little strange about the results. If I run top from a shell in my container then I get the information from the container’s point of view (POV):

Note that the shell and top have PIDs of 1 and 26 respectively. Now look at the output from the top query of the Docker Remote API for that container (after removing the header):

{“Processes”:[

[“root”,”3392″,”1354″,”0″,”09:26″,”pts/1″,”00:00:00″,”/bin/bash”],

[“root”,”4794″,”3392″,”0″,”10:18″,”pts/1″,”00:00:00″,”top”]],

“Titles”:[“UID”,”PID”,”PPID”,”C”,”STIME”,”TTY”,”TIME”,”CMD”]}

Now it says that shell and top have PIDs of 3392 and 4794 respectively. These PIDs come from the host system’s POV. Threat Stack maps one POV to the other, allowing you to understand what the container is actually doing and whether your workload is reaching outside of its container.

Threat Stack does not hook into the kernel and put itself in between your process and execution. Instead, it leverages the Linux Audit System Architecture to subscribe to security events sent out from the kernel. This means that Threat Stack will still catch all security events, even while some of those events might not have completely populated fields. Our Docker integration uses transient information from the operating system about process details. The Threat Stack agent will still report the event without container information if the process terminates before we can augment the event with container values.

Let’s move to a more complete example. We still have container 034fe81c01e1 and I already have Threat Stack Pro Edition running on my machine. Next I enable the support for Docker containers. For those following along at home, please contact Threat Stack to receive instructions on how to do this in your deployment.

Now let’s go into that first container and install curl with a simple apt-get command. We can then see what it looks like when I use curl to download some malware that enables lateral movement on the network.

Threat Stack sees that somebody on a container just pulled something down from a Chinese domain. We do not use any repositories from China in our product, so that warrants some looking into!

Suppose somebody a bit nefarious wants to steal data from in that container, or use Docker to give them a privileged shell. I would like to know about that. So let’s simulate that activity by logging into the container.

$ docker exec -i -t some-nginx bash

Being an unsophisticated insider of questionable morals, I realize I do not have my trusty scp tool to get the data to my offsite account. That’s easy enough to rectify with apt-get, but then Threat Stack will notice something fishy starting to happen.

At this point IT gets an alert and comes running down the hall to bang on my door and tell me I’m in big trouble. They also call law enforcement and point them toward competitor.com. Threat Stack brings visibility to the system. For comparison, here’s what the same action looks like when not attributed to a container:

This just scratches the surface of the Docker integration. “Docker use is exploding and the correlation of a host-based intrusion detection system (HIDS) with Docker and AWS CloudTrail is what gives Docker users the vital context they need to act swiftly on a security incident,” said Venkat Pothamsetty, Threat Stack’s vice president of products. “This integration is what makes the Threat Stack Pro Edition solution unique and gives customers operating on AWS with Docker additional information to evaluate their security strategy, to ensure all activity within their application environment is secure and compliant.” We’re happy to show you more, and we’re looking for feedback about what you’d like to see in the future. See what you’ve been missing!

Nathan Cooprider is a Senior Software Engineer working on the Threat Stack instance agent. Nathan comes to Threat Stack from the endpoint engineering team of Bit9 + Carbon Black. Prior to Bit9, Nathan led the signal processing software team for the MQ9 Predator drone at BAE. He received his BS in CS from Brigham Young University and his PhD in CS from the University of Utah. Nathan has over a decade of experience working with computer systems. This includes eight refereed publications on the static analysis of microcontroller applications written in C. He also wrote a paper on multivariate data visualization, co-authored a paper on multiple hypothesis tracking, and has supported language modeling research. Nathan's accumulated experience with various software engineering languages and tools includes C, C++, python, doxygen, Jenkins, OCaml, CIL, cmake, and many others.