Docker is an open-source project that automates the deployment of applications inside software containers.
The DockerLatentWorker attempts to instantiate a fresh image for each build to assure consistency of the environment between builds.
Each image will be discarded once the worker finished processing the build queue (i.e. becomes idle).
See build_wait_timeout to change this behavior.

Beside, it is always possible to install Docker next to the buildmaster.
Beware that in this case, overall performance will depend on how many builds the computer where you have your buildmaster can handle as everything will happen on the same one.

Note

It is not necessary to install Docker in the same environment as your master as we will make use to the Docker API through docker-py.
More in master setup.

CoreOS is targeted at building infrastructure and distributed systems.
In order to get the latent worker working with CoreOS, it is necessary to expose the docker socket outside of the Virtual Machine.
If you installed it via Vagrant, it is also necessary to uncomment the following line in your config.rb file:

$expose_docker_tcp=2375

The following command should allow you to confirm that your Docker socket is now available via the network:

boot2docker is one of the fastest ways to boot to Docker.
As it is meant to be used from outside of the Virtual Machine, the socket is already exposed.
Please follow the installation instructions on how to find the address of your socket.

Our build master will need the name of an image to perform its builds.
Each time a new build will be requested, the same base image will be used again and again, actually discarding the result of the previous build.
If you need some persistent storage between builds, you can use Volumes.

Each Docker image has a single purpose.
Our worker image will be running a buildbot worker.

Docker uses Dockerfiles to describe the steps necessary to build an image.
The following example will build a minimal worker.
This example is voluntarily simplistic, and should probably not be used in production, see next paragraph.

We will rely on docker-py to connect our master with docker.
Now is the time to install it in your master environment.

Before adding the worker to your master configuration, it is possible to validate the previous steps by starting the newly created image interactively.
To do this, enter the following lines in a Python prompt where docker-py is installed:

It is now time to add the new worker to the master configuration under workers.

The following example will add a Docker latent worker for docker running at the following address: tcp://localhost:2375, the worker name will be docker, its password: password, and the base image name will be my_project_worker:

(mandatory)
The worker password part of the Latent Workers API.
If the password is None, then it will be automatically generated from random number, and transmitted to the container via environment variable.

In addition to the arguments available for any Latent Workers, DockerLatentWorker will accept the following extra ones:

docker_host

(mandatory)
This is the address the master will use to connect with a running Docker instance.

image

This is the name of the image that will be started by the build master. It should start a worker.
This option can be a renderable, like Interpolate, so that it generates from the build request properties.

command

(optional)
This will override the command setup during image creation.

(optional if image is given)
This is the content of the Dockerfile that will be used to build the specified image if the image is not found by Docker.
It should be a multiline string.

Note

In case image and dockerfile are given, no attempt is made to compare the image with the content of the Dockerfile parameter if the image is found.

version

(optional, default to the highest version known by docker-py)
This will indicates which API version must be used to communicate with Docker.

tls

(optional)
This allow to use TLS when connecting with the Docker socket.
This should be a docker.tls.TLSConfig object.
See docker-py’s own documentation for more details on how to initialise this object.

(optional, defaults to socket.getfqdn())
Address of the master the worker should connect to. Use if you master machine does not have proper fqdn.
This value is passed to the docker image via environment variable BUILDMASTER

The volume parameter allows to share directory between containers, or between a container and the host system.
Refer to Docker documentation for more information about Volumes.

The format of that variable has to be an array of string.
Each string specify a volume in the following format: volumename:bindname.
The volume name has to be appended with :ro if the volume should be mounted read-only.

Note

This is the same format as when specifying volumes on the command line for docker’s own -v option.

The HyperLatentWorker attempts to instantiate a fresh image for each build to assure consistency of the environment between builds.
Each image will be discarded once the worker finished processing the build queue (i.e. becomes idle).
See build_wait_timeout to change this behavior.

In addition to the arguments available for any Latent Workers, HyperLatentWorker will accept the following extra ones:

password

(mandatory)
The worker password part of the Latent Workers API.
If the password is None, then it will be automatically generated from random number, and transmitted to the container via environment variable.

hyper_host

(mandatory)
This is the address the hyper infra endpoint will use to start docker containers.

image

(mandatory)
This is the name of the image that will be started by the build master. It should start a worker.
This option can be a renderable, like Interpolate, so that it generates from the build request properties.
Images are by default pulled from the public DockerHub docker registry.
You can consult the hyper documentation to know how to configure a custom registry.
HyperLatentWorker does not support starting a worker built from a Dockerfile.

masterFQDN

(optional, defaults to socket.getfqdn())
Address of the master the worker should connect to. Use if you master machine does not have proper fqdn.
This value is passed to the docker image via environment variable BUILDMASTER

If the value contains a colon (:), then BUILDMASTER and BUILDMASTER_PORT environment variables will be passed, following scheme: masterFQDN="$BUILDMASTER:$BUILDMASTER_PORT"

This feature is useful for testing behind a proxy using ngrok command like: ngroktcp9989ngrok config can the be retrieved with following snippet:

The MarathonLatentWorker attempts to instantiate a fresh image for each build to assure consistency of the environment between builds.
Each image will be discarded once the worker finished processing the build queue (i.e. becomes idle).
See build_wait_timeout to change this behavior.

In addition to the arguments available for any Latent Workers, MarathonLatentWorker will accept the following extra ones:

marathon_url

(mandatory)
This is the URL to Marathon server.
Its REST API will be used to start docker containers.

(mandatory)
This is the name of the image that will be started by the build master. It should start a worker.
This option can be a renderable, like Interpolate, so that it generates from the build request properties.
Images are by pulled from the default docker registry.
MarathonLatentWorker does not support starting a worker built from a Dockerfile.

masterFQDN

(optional, defaults to socket.getfqdn())
Address of the master the worker should connect to. Use if you master machine does not have proper fqdn.
This value is passed to the docker image via environment variable BUILDMASTER

If the value contains a colon (:), then BUILDMASTER and BUILDMASTER_PORT environment variables will be passed, following scheme: masterFQDN="$BUILDMASTER:$BUILDMASTER_PORT"

marathon_extra_config

(optional, defaults to {}`)
Extra configuration to be passed to Marathon API.
This implementation will setup the minimal configuration to run a worker (docker image, BRIDGED network)
It will let the default for everything else, including memory size, volume mounting, etc.
This configuration is voluntarily very raw so that it is easy to use new marathon features.
This dictionary will be merged into the Buildbot generated config, and recursively override it.
See Marathon API documentation to learn what to include in this config.