Let’s say you want to run a PHP application like WordPress under Apache. Here are the things you need to keep secure:

WordPress itself

All plugins, themes, customizations

All PHP libraries it uses (MySQL, image-processing, etc.)

MySQL

Apache

All libraries MySQL or Apache use: OpenSSL, libc, PHP itself, etc.

The kernel

All containerization tools

On Debian (and most of its best-known derivatives), we are extremely lucky to have a wonderful security support system. If you run a Debian system, the combination of unattended-updates, needrestart, debsecan, and debian-security-support will help one keep a Debian system secure and verify it is. When the latest OpenSSL bug comes out, generally speaking by the time I wake up, unattended-updates has already patched it, needrestart has already restarted any server that uses it, and I’m protected. Debian’s security team generally backports fixes rather than just say “here’s the new version”, making it very safe to automatically apply patches. As long as I use what’s in Debian stable, all layers mentioned above will be protected using this scheme.

This picture is much nicer than what we see in Docker.

Problems

We have a lot of problems in the Docker ecosystem:

No built-in way to know when a base needs to be updated, or to automatically update it

Diverse and complicated vendor security picture

No way to detect when intermediate libraries need to be updated

Complicated final application security picture

Let’s look at them individually.

Problem #1: No built-in way to know when a base needs to be updated, or to automatically update it

First of all, there is nothing in Docker like unattended-updates. Although a few people have suggested ways to run unattended-updates inside containers, there are many reasons that approach doesn’t work well. The standard advice is to update/rebuild containers.

So how do you know when to do that? It is not all that obvious. Theoretically, official OS base images will be updated when needed, and then other Docker hub images will detect the base update and be rebuilt. So, if a bug in a base image is found, and if the vendors work properly, and if you are somehow watching, then you could be protected. There is work in this area; tools such as watchtower help here.

But this can lead to a false sense of security, because:

Problem #2: Diverse and complicated vendor security picture

Different images can use different operating system bases. Consider just these official images, and the bases they use: (tracking latest tag on each)

nginx: debian:stretch-slim (stretch is pre-release at this date!)

mysql: debian:jessie

mongo: debian:wheezy-slim (previous release)

apache httpd: debian:jessie-backports

postgres: debian:jessie

node: buildpack-deps:jessie, eventually depends on debian:jessie

wordpress: php:5.6-apache, eventually depends on debian:jessie

And how about a few unofficial images?

oracle/openjdk: oraclelinux:latest

robotamer/citadel: debian:testing (dangerous, because testing is an alias for different distros at different times)

docker.elastic.co/kibana: ubuntu of some sort

The good news is that Debian jessie seems to be pretty popular here. The bad news is that you see everything from Oracle Linux, to Ubuntu, to Debian testing, to Debian oldstable in just this list. Go a little further, and you’ll see Alpine Linux, CentOS, and many more represented.

Here’s the question: what do you know about the security practices of each of these organizations? How well updated are their base images? Even if it’s Debian, how well updated is, for instance, the oldstable or the testing image?

The attack surface here is a lot larger than if you were just using a single OS. But wait, it gets worse:

Problem #3: No way to detect when intermediate libraries need to be updated

Let’s say your Docker image is using a base that is updated immediately when a security problem is found. Let’s further assume that your software package (WordPress, MySQL, whatever) is also being updated.

What about the intermediate dependencies? Let’s look at the build process for nginx. The Dockerfile for it begins with Debian:stretch-slim. But then it does a natural thing: it runs an apt-get install, pulling in packages from both Debian and an nginx repo.

I ran the docker build across this. Of course, the apt-get command brings in not just the specified packages, but also their dependencies. Here are the ones nginx brought in:

Now, what is going to trigger a rebuild if there’s a security fix to libssl1.1 or libicu57? (Both of these have a history of security holes.) The answer, for the vast majority of Docker images, seems to be: nothing automatic.

Problem #4: Complicated final application security picture

And that brings us to the last problem: Let’s say you want to run an application in Docker. exim, PostgreSQL, Drupal, or maybe something more obscure. Who is watching for security holes in it? If you’re using Debian packages, the Debian security team is. If you’re using a Docker image, well, maybe it’s the random person that contributed it, maybe it’s the vendor, maybe it’s Docker, maybe it’s nobody. You have to take this burden on yourself, to validate the security support picture for each image you use.

Conclusion

All this adds up to a lot of work, which is not taken care of for you by default in Docker. It is no surprise that many Docker images are insecure, given this picture. The unfortunate reality is that many Docker containers are running with known vulnerabilities that have known fixes, but just aren’t, and that’s sad.

I wonder if there are any practices people are using that can mitigate this better than what the current best-practice recommendations seem to be?

Post navigation

6 thoughts on “Is there any way to truly secure Docker container contents?”

I really liked your blog entry, as it is showing the real problem of this container stuff: “Make it easy for the developers” means roll out a fixed snapshot of some “old crap”, yeah thats making deployment much easier, but only because you are removing the security aspect we have learned over the years. And this learned us the hard way,why we are needing good security teams.

Much the same problem exists in the set of languages-with-deploy-systems – python, Ruby, Go, Rust – where if you are lucky you have a set of tools to tell you the dependencies for a particular application, and if you are particularly lucky you can reproduce a build with the exact version dependencies.

Say that Go’s SSL implementation has an issue. For packages that your distro provided, you probably have a built-in dependency graph – but Go is always built statically, so it isn’t a matter of replacing Go-libssl (or whatever it’s called) and restarting your apps. You need a new build of each and every application.

Its reports are rather basic and it has nothing to do with rebuilding images. Also, it will scan all packages inside an image, even if the intended application process would not link or use them. So you may get some false positives which do not concern your application (e.g. a coreutils vulnerability would not affect nginx in the same container).