thnee.se

Docker builder pattern

August 19, 2018

When talking about Docker, the typical case is that the resulting build artifact of
the project is a Docker image that will be pushed to a Docker registry
and subsequently run on some production server.

That’s not what this blog post is about.

Docker can be used very effectively in a project, to create isolated build environments,
even if the result of the project has nothing to do with Docker or containers.

Let’s create a sample project that is a Python library called superheroes,
where the resulting build artifact is a cheese wheel that will be uploaded to PyPI.
(Using Python is just an example that I am familiar with,
the Docker patterns can be applied to any programming language).

This project is just some random library,
it has nothing to do with Docker or containers in and of itself,
but Docker can be used to create an isolated and reproducable environment
that is used to run the tests, build the package, and run the package uploading program.

This post will not go into all the details of how to write and test a Python package,
just keep in mind that the goal is to take what is in the src/ directory,
and produce build artifacts into a dist/ directory,
and upload those build artifacts to PyPI.

But what I do want to hilight are the Dockerfile and build.sh files.

Let’s use a couple of different formatting and testing tools
to showcase the Docker usage.

Tox runs the python tests.

Shellcheck is used to validate the formatting of build.sh.

Hadolint is used to validate the formatting of Dockerfile.

Black is used to do automatic formatting of the Python source code.

Beautysh is used to do automatic formatting of build.sh.

Dockerfile

Let’s begin with the Dockerfile that is used to build the builder image.
The resulting builder image is used as a starting point for all build related commands,
so it contains requirements both for the tests and for the automatic code formatters.

Running the tests

$ ./build.sh run-tests

First, it builds an updated vresion of the builder image, or exit on failure.

Exiting on failure is important because if building the builder image fails,
but there is an old builder image laying around, the old one will be used instead,
which might result in a false positive outcome of the test run.

Then, it runs the three test suites that we have chosen for this project,
Tox, Shellcheck, and Hadolint.
Again, exiting on failure for each step, so that any errors are immediately visible,
and not hidden further up in the output.

Finally, it runs the signal handler that was attached earlier,
that cleans up all builder containers that have been started.

Automatic code formatting

$ ./build.sh format-code

First, it builds an updated version of the builder image.

Then, it runs black which is the chosen code formatter for this project,
Note that the source code files are modified in place by black,
and since black is running in the builder container,
the updated files are also written inside the builder container,
so they are not immediately available anywhere on the host machine yet.

At this point the builder process is finished and the container status is Exited.
But the container is still perfectly intact and remains in Docker.
It can be seen by calling the docker ps -a command.
All the files that were modified are still available
in the file system of the exited container.

Then, it copies the updated source code files from the builder container,
which will overwrite the source code files in the working directory on the host machine.

The process is repeated for the next code formatting tool, beautysh.

Finally, it cleans up all builder containers via the signal handler.

Building the package

$ ./build.sh build-package

First, it builds an updated version of the builder image.

Then, it runs the python specific build script setup.py sdist bdist_wheel,
which will produce the two build artifacts in dist/ (a .whl file and a .tar.gz file).
Note that the dist/ folder is created inside the builder container,
so it is not immediately available anywhere on host machine.
The container is now exited, but remains intact in docker,
just like in the previous step.

Then, it commits the container as a new image.
This will take everything from the run, and save it as a new version of the image,
including the dist/ folder that was just created.

Finally, it cleans up all builder containers via the signal handler.

Uploading the package

Assuming that this command is called after build-package,
the builder image will now contain the dist/ folder with the build artifacts in it.
(Here it is up to the user to determine if the build-package step succeeded or not
before running this command).

So, instead of building an updated version of the builder image,
it assumes that there already is a builder image available from before,
and uses that to run the upload command.

The python specific program twine uploads the build artifacts
from the dist/ folder to PyPI.