Dray: UNIX pipes for Docker

Brian DeHamer

March 12, 2015

Share +

Most of the Docker use cases involve the use of containers for hosting long-running services. These are things like a web application, database or message queue -- services that are running continuously, waiting to service requests. Another interesting use case that we've been playing with here at CenturyLink Labs is leveraging Docker to execute short-lived, single-purpose tasks (in fact our own Ross Jimenez published an article titled Using Ephermeral Containers in a Micro-Services Workflow a few weeks back that discusses this exact thing). We've published a few of these Docker-wrapped tasks in the past:

(Edit: Updated after release of imagelayers.io)imagelayers.io - displays the layers that compose your Docker image pulled from the Docker Hub or a private Docker registry.

All of the benefits that apply to long-running services (portability, isolation, etc...) also apply when wrapping one-off tasks in Docker images. Both of the examples above happen to have been implemented in Ruby but the fact that they're wrapped in Docker images means that no one has to worry about running the correct version of the Ruby interpreter or installing our various dependencies. If you've got Docker, you can execute either of these utilities with a simple docker run. Once you've started to wrap tasks like these in containers you may find it useful to string them together and have the output of one container feed the input of the next container. Something like the capability provided by Unix pipes:

The example above is pretty simplistic but it describes a job which consists of three steps. Each step references the name of a Docker image to be executed. When receiving this job description, Dray will start a container from the "centurylink/randowrd" image. As the container is executing Dray will capture any data written to the container's stdout stream so that it can be passed along to the next step in the list. Once the "randword" container exits, Dray will start the "centurylink/upper" container and pass any data captured in the previous step to that container's stdin stream. Dray will continue executing each of the steps in this manner, marshalling the stdout of one step to the stdin of the next step, until all of the steps have been completed. The log output from Dray for the job shown above would look something like this:

In between all the container management messages can see the the output that is generated by the containers themselves. The "randword" container outputs a string to stdout which is picked-up by Dray and passed to the "upper" container. The output from "upper" is an upper-cased version of the string that it received as input -- this is captured by Dray and fed into the "reverse" container which simply reverses the string. Again, this is a fairly useless example, but it serves to illustrate the data marshalling that Dray provides.

Real Use Cases

We've been using the Dray engine for a little while now to power some of the features in our Panamax project. One of the newer features allows users to provision server clusters on a number of different cloud providers and automatically install the Panamax agent software. This provisioning process requires a number of different steps including the provisioning of servers, the configuration of networks and installation of software. Some of these steps are provider-specific while others are provider-agnostic. The approach we took was to implement each of the discrete steps as an image-wrapped task and then create a set of Dray jobs that composed different combinations of these tasks for the different cloud-providers.

The Dray engine coordinates the execution of the jobs and handles the marshalling of data from one step to the next (e.g. the IP addresses of the servers created in one step need to be passed to the step that does the software installation). This is just one application of Dray, but we suspect that there may be other interesting ways to use it. If you want to give it a spin, you can pull the centurylink/dray image from the Docker hub. This is just our first release and aren't entirely sure where we're going to take it next, but we'd love to hear from you if you come up with an interesting use case or want to contribute to the code (it's all available on GitHub).