LXC-based Builder Configuration

CircleCI Enterprise builder behavior can be customized with a number parameters specified via environment variables. These variables can be specified by customizing the startup script (or *.tf file if using Terraform) for builders to the following instead of the default:

Adjusting Build Container Power

You can, optionally, adjust the CPU and memory used by your build containers using the following build configuration variables. Our default containers have 2 CPUs and 4G of memory:

CIRCLE_CONTAINER_MEMORY_LIMIT - the amount of memory allocated to each container set with a value of xG where x is an integer number of gigabytes of memory.
Default value is 4G.

CIRCLE_CONTAINER_CPUS - the number of CPUs allocated to each container.
Default value is 2.
Note: if CIRCLE_CONTAINER_CPUS is 0 there will be no limit on cpu resources per container.

CIRCLE_NUM_CONTAINERS - the number of containers to run on each build box.
Default value is ([TOTAL_CORES] - 2)/CIRCLE_CONTAINER_CPUSNote:You must leave 2 cores and at least 4G of memory free to run the Circle process in the container.

In order to set them up, please follow the following steps:

Add them to the user data field of the Launch Configuration associated with your auto-scaling group in amazon.

Terminate all instances in the auto-scaling group. If you are worried about downtime, you can launch new instances first before terminating the old instances.

Selecting a build container image

You can control the container image used to run builds with the CIRCLE_CONTAINER_IMAGE_URI environment variable. The URI may either be a standard http(s) resource, or it may use the s3:// protocol (e.g. s3://bucket-name/key) in which case AWS IAM instance profile authentication may be used. For CircleCI-managed containers, you may also use the abbreviated, region-independent CIRCLE_CONTAINER_IMAGE_ID variable to fix your builder machines to a specific container version.

Using the Trusty image

CircleCI builders can use the same Trusty (Ubuntu 14.04) build container image available on circleci.com by specifying CIRCLE_CONTAINER_IMAGE_ID=circleci-trusty-container_0.0.575. This image includes the languages and packages specified here.

Using a custom image

If you’re running into serious performance issues configuring your dependencies via the circle.yml, CircleCI enterprise has functionality that allows you to generate your own custom image and use it to run all your builds. This can often lead to much faster startup times, since the new container will come pre-configured with all your dependencies, and is frequently much smaller than our default container. Some things to consider before deciding if this is the right approach for you:

Once you move to a custom image container, you will be responsible for all future updates to your own container.

If you are interested, please reach out to enterprise-support@circleci.com as well. We’d love to hear your feedback and give guidance on how best to configure your build environment

Running Upstream Docker

If you wish to use the upstream version of Docker rather than the patched version provided by CircleCI you will need to run an privileged LXC container. You can do this by editing the launch configuration in AWS and setting CIRCLE_CONTAINER_TYPE=privileged-lxc.

You can now either bake the version of Docker that you want into your custom image, or install it via circle.yml by adding the code below:

Sharing Docker Socket

In CircleCI Enterprise, it’s possible to bind-mount a Docker socket from the underlying host into the build containers so that they can all share a single long-running daemon for improved performance. It should be equivalent to the cache performance you see on Jenkins.

NOTE: The primary drawback here is security-related. Giving builds access to the Docker daemon on the host is basically equivalent to giving them root access to the machine. The same is true on Jenkins, but CircleCI’s default nested Docker approach, while it has performance issues, does provide a layer of security.

To setup a long-running Docker daemon on a builder machine, you can start it with userdata like this:

There are also a couple small differences to use this daemon within builds:

Set DOCKER_HOST="unix:///tmp/docker.sock"

Don’t enter “docker” in the “services” section of circle.yml (no need to start the service as it’s already running on the host)

You should then see improved performance between builds!

Adjusting Builder Networking

You can adjust which subnets containers use by setting the following environment variables:

CIRCLE_CONTAINERS_SUBNET - the subnet all containers will share, in CIDR notation.

CIRCLE_CONTAINERS_SUBNET_MASK_LENGTH - the size of the netmask for
each container’s individual subnet. Needs to be no more than 30, and needs to be a large
enough to fit enough containers in the CIRCLE_CONTAINERS_SUBNET.