Data Science Deployments with Docker

Madison May -
November 15, 2016

Deploying machine learning models has always been a struggle. Most of the software industry has adopted the use of container engines like Docker for deploying code to production, but since accessing hardware resources like GPUs from Docker was difficult and required hacky, driver specific workarounds, the machine learning community has shied away from this option. With the recent release of NVIDIA’s nvidia-docker tool, however, accessing GPUs from within Docker is a breeze, and we’re already reaping the benefits here at indico. In this tutorial we’ll walk you through setting up nvidia-docker so you too can deploy machine learning models with ease.

Before we get into the details however, let’s talk briefly about why using Docker for your next data science project may be a good choice. There is certainly a learning curve for the tools in the Docker ecosystem, but the benefits are worth the effort.

No inconsistencies between team environment configurations:

Software configuration is always a pain. Docker’s configure once, run anywhere model means your teammates will have to worry less about environment setup and can focus more on writing code and building machine learning models.

Reliable deployments:

Fewer bugs crop up in production when you can be assured that your development environment is identical to your production environment.

Git-like tool for environment configuration:

If something does go wrong in production, reverting to a previous Docker image ensures you can quickly get back to a functional state.

Why is a special solution needed for using GPUs within Docker?

Docker is designed to be hardware and platform agnostic. GPUs are specialized hardware that is not necessarily available on every host. Because of this, the Docker binary does not include GPU support out of the box, and requires a fair amount of configuration to get things working properly. When we first started using Docker in production and needed to enable access to GPU devices from within the container, we had to roll our own solution. It was educational to have to understand the mechanisms by which hardware like GPUs are exposed to an operating system (primarily the /dev block), but we ended up with a solution that was not portable and required that the host’s NVIDIA driver was identical to a second copy of the driver installed within the container. Whenever we updated our NVIDIA drivers to support newer CUDA versions, we had to make a breaking change to our Docker image in order to ensure drivers matched exactly.

Thankfully, the nice folks at NVIDIA have rectified this problem by releasing nvidia-docker, a tool for configuring docker to allow GPU access from within containers.

How does nvidia-docker work?

nvidia-docker takes the following steps to get CUDA working within your container:

This means that as long as you have a functional NVIDIA driver on your host and a CUDA version recent enough to support your driver is installed within your container, you should be able to execute CUDA code from your running Docker container. Importantly, the Docker container can also be run in another environment with different driver versions, making it easy to build once and then run anywhere.

Alex has already done the hard work of providing us with a Docker image of his project, and has gone to the trouble of installing the necessary CUDA drivers in the Docker image as well. Normally we’d need to have a functioning installation of CUDA, Theano, and the lasagne library in order to run his code, but since he’s provided us with a Docker image we should be up and running in just a few minutes.

Hey, this isn’t so bad. The dockerfile Alex used is based off of an official NVIDIA Docker image (nvidia/cuda:7.5-cudnn4-devel) that already includes the required CUDA libraries, so it only has to describe how to install a few system dependencies for working with image formats, install a few machine learning Python packages with pip (Theano, lasagne, etc.), and download some pre-trained model weights. It’s little more than a glorified bash setup script.

The doodle alias isn’t bad either. It simply specifies the Docker image we’ll be running (alexjc/neural-doodle:gpu) and lets Docker know that the ./samples and ./frames directories should be accessible from the Docker container at /nd/samples/ and /nd/frames. This is done using Docker’s “volumes” feature, which the curious can read more about on the official Docker site.

At indico, we now use a setup to the neural-doodle configuration to host the indico API on Amazon GPUs. Instead of using our own bash scripts, we allow the nvidia-docker tool to handle the process of ensuring device drivers within the Docker container match device drivers on the host. This means when our customers wish to run our APIs on their local machines, deployment is as easy as providing them with access to our production Docker image and letting the nvidia-docker tool handle the rest.

Operating System Support

At the moment, nvidia-docker is only portable in the sense that it’s not reliant on a particular GPU model, NVIDIA driver version, or linux distribution. Running nvidia-docker on OSX or Windows will likely not be supported anytime soon.

Where can I find more information on nvidia-docker?

NVIDIA has done an excellent job of keeping the wiki of their Github page up-to-date. Chances are if you have questions that aren’t answered in this blog post, you can probably find answers in the nvidia-dockerGithub wiki.

If you’re using a version of CUDA other than the one used in this demo (CUDA 7.5), you might also want to take a peek at the full list of base images that NVIDIA provides for you to work with.

I hope you’ve enjoyed this whirlwind tour on using nvidia-docker to build and run machine learning projects, and perhaps created a bit of original algorithmic art while you’re at it. If you run into trouble trying out this tutorial, or want to learn more about how we’re using Docker in production at indico, feel free to reach out over our site chat and say hello. Happy hacking!