How to autoscale continuous deployment with GitLab Runner on DigitalOcean

GitLab CI/CD is an effective way to build the habit of testing all code before it’s deployed. GitLab CI/CD is also highly scalable thanks to an additional tool, GitLab Runner, which automates scaling your build queue in order to avoid long wait times for development teams trying to release code.

In this guide, we will demonstrate how to configure a highly scalable GitLab infrastructure that manages its own costs, and automatically responds to load by increasing and decreasing available server capacity.

Goals

We’re going to build a scalable CI/CD process on DigitalOcean that automatically responds to demand by creating new servers on the platform and destroys them when the queue is empty.

These reusable servers are spawned by the GitLab Runner process and are automatically deleted when no jobs are running, reducing costs and administration overhead for your team.

As we’ll explain in this tutorial, you are in control of how many machines are created at any given time, as well as the length of time they’re retained before being destroyed.

We’ll be using three separate servers to build this project, so let’s go over terminology first:

GitLab Bastion: The bastion server or Droplet is the core of what we’ll be configuring. It is the control instance that is used to interact with the DigitalOcean API to create Droplets and destroy them when necessary. No jobs are executed on this server.

GitLab Runner: Your runners are transient servers or Droplets that are created on the fly by the bastion server when needed to execute a CI/CD job in your build queue. These servers are disposable, and are where your code is executed or tested before your build is marked as passing or failing.

By leveraging each of the GitLab components, the CI/CD process will enable you to scale responsively based on demands. With these goals in mind, we are ready to begin setting up our continuous deployment with GitLab and DigitalOcean.

Prerequisites

This tutorial will assume you have already configured GitLab on your own server or through the hosted service, and that you have an existing DigitalOcean account.

Step 2: Set up infrastructure

We will create two types of machines in this tutorial: a bastion instance, which controls and spawns new machines, and our runner instances, which are temporary servers spawned by the bastion Droplet to build code when required. The bastion instance uses Docker to create your runners.

Here are the DigitalOcean products we’ll use, and what each component is used for:

Flexible Droplets — We will create memory-optimized Droplets for our GitLab Runners as it’s a memory-intensive process which will run using Docker for containerization. You can shrink or grow this Droplet in the future as needed, however we recommend the flexible Droplet option as a starting point to understand how your pipeline will perform under load.

DigitalOcean Spaces (Object Storage) — We will use DigitalOcean Spaces to persist cached build components across your runners as they’re created and destroyed. This reduces the time required to set up a new runner when the CI pipeline is busy, and allows new runners to pick up where others left off immediately.

To start, we’ll create the bastion Droplet. Create a new Droplet, then under choose an image, select the One-click apps tab. From there, select Docker 17.12.0-ce on 16.04 (note that this version is current at the time of writing), then choose the smallest Droplet size available, as our bastion Droplet will manage the creation of other Droplets rather than actually perform tests.

It is recommended that you create your server in a data center that includes DigitalOcean Spaces in order to use the object storage caching features mentioned earlier.

Select both the Private networking and Monitoring options, then click Create Droplet.

We also need to set up our storage space which will be used for caching. Follow the steps in “How To Create a DigitalOcean Space and API Key” to create a new Space in the same or nearest data center as your hosted GitLab instance, along with an API Key.

Note this key down, as we’ll need it later in the tutorial.

Now it’s time to get our CI started!

Step 3: Configure the GitLab Runner Bastion Server

With the fresh Droplet ready, we can now configure GitLab Runner. We’ll be installing scripts from GitLab and GitHub repositories.

As a best practice, be sure to inspect scripts to confirm what you will be installing prior to running the full commands below.

With these installations complete, we can move on to connecting our GitLab Runner to our GitLab install.

Step 4: Obtain Runner registration token

To link GitLab Runner to your existing GitLab install, we need to link the two instances together by obtaining a token that authenticates your runner to your code repositories.

Login to your existing GitLab instance as the admin user, then click the wrench icon to enter the admin settings area.

On the left of your screen, hover over Overview and select Runners from the list that appears.

On the Runners page under the How to set up a shared Runner for a new project section, copy the token shown in Step 3, and make a note of it along with the publicly accessible URL of your GitLab instance from Step 2. If you are using HTTPS for Gitlab, make sure it is not a self-signed certificate, or GitLab Runner will fail to start.

Step 5: Configure GitLab on the Bastion Droplet

Back in your SSH connection with your bastion Droplet, run the following command:

sudo gitlab-runner register

This will initiate the linking process, and you will be asked a series of questions.

On the next step, enter the GitLab instance URL from the previous step:

Choose whether or not your runner should be able to run untagged jobs. This setting allows you to choose whether your runner should build repositories with no tags at all, or require specific tags. Select true in this case, so your runner can execute all repositories.

Whether to run untagged jobs[true/false]: true

Choose if this runner should be shared among your projects, or locked to the current one, which blocks it from building any code other than those specified. Select false for now, as this can be changed later in GitLab’s interface:

Whether to lock Runner to current project [true/false]: false

Choose the executor which will build your machines. Because we’ll be creating new Droplets using Docker, we’ll choose docker+machine here, but you can read more about the advantages of each approach in this compatibility chart:

With Docker Machine upgraded, we can move on to setting up our access tokens for GitLab Runner to use.

Step 7: Gather DigitalOcean credentials

Now we need to create the credentials that GitLab Runner will use to create new Droplets using your DigitalOcean account.

Visit your DigitalOcean dashboard and click API. On the next screen, look for Personal access tokens and click Generate New Token.

Give the new token a name you will recognize such as GitLab Runner Access and ensure that both the read and write scopes are enabled, as we need the Droplet to create new machines without human intervention.

Copy the token somewhere safe as we’ll use it in the next step. You can’t retrieve this token again without regenerating it, so be sure it’s stored securely.

Step 8: Edit GitLab Runner configuration files

To bring all of these components together, we need to finish configuring our bastion Droplet to communicate with your DigitalOcean account.

In your SSH connection to your bastion Droplet, use your favorite text editor, such as nano, to open the GitLab Runner configuration file for editing:

nano /etc/gitlab-runner/config.toml

This configuration file is responsible for the rules your CI setup uses to scale up and down on demand. To configure the bastion to autoscale on demand, you need to add the following lines:

concurrent = 50 # All registered Runners can run up to 50 concurrent builds[[runners]]
url ="https://example.digitalocean.com"
token ="existinggitlabtoken"# Note this is different from the registration token used by `gitlab-runner register`
name ="example-runner"
executor ="docker+machine"# This Runner is using the 'docker+machine' executor
limit = 10 # This Runner can execute up to 10 builds (created machines)[runners.docker]
image ="alpine:latest"# Our secure image[runners.machine]
IdleCount = 1 # The amount of idle machines we require for CI if build queue is empty
IdleTime = 600 # Each machine can be idle for up to 600 seconds, then destroyed
MachineName ="gitlab-runner-autoscale-%s"# Each machine will have a unique name ('%s' is required and generates a random number)
MachineDriver ="digitalocean"# Docker Machine is using the 'digitalocean' driver
MachineOptions =["digitalocean-image=coreos-stable", # The DigitalOcean system image to use by default"digitalocean-ssh-user=core", # The default SSH user"digitalocean-access-token=DO_ACCESS_TOKEN", # Access token from Step 7"digitalocean-region=nyc3", # The data center to spawn runners in"digitalocean-size=1gb", # The size (and price category) of your spawned runners"digitalocean-private-networking"# Enable private networking on runners][runners.cache]
Type ="s3"# The Runner is using a distributed cache with the S3-compatible Spaces service
ServerAddress ="nyc3.spaces.digitaloceanspaces.com"
AccessKey ="YOUR_SPACES_KEY"
SecretKey ="YOUR_SPACES_SECRET"
BucketName ="your_bucket_name"
Insecure =true# We do not have a SSL certificate, as we are only running locally

Once you’ve added the new lines, customize the access token, region and Droplet size based on your setup. For the purposes of this tutorial, we’ve used the smallest Droplet size of 1GB and created our Droplets in NYC3. Be sure to use the information that is relevant in your case.

You also need to customize the cache component, and enter your Space’s server address from the infrastructure configuration step, access key, secret key and the name of the Space that you created.

When completed, restart GitLab Runner to make sure the configuration is being used:

Step 9 — Test Your GitLab Runner

At this point, our GitLab Runner bastion Droplet is configured and is able to create DigitalOcean Droplets on demand, as the CI queue fills up. We’ll need to test it to be sure it works by heading to your GitLab instance and the project we imported in Step 1.

To trigger a build, edit the readme.md file by clicking on it, then clicking edit, and add any relevant testing text to the file, then click Commit changes.

Now a build will be automatically triggered, which can be found under the project’s CI/CD option in the left navigation.

On this page you should see a pipeline entry with the status of running. In your DigitalOcean account, you’ll see a number of Droplets automatically created by GitLab Runner in order to build this change.

Congratulations! Your CI pipeline is cloud scalable and now manages its own resource usage. After the specified idle time, the machines should be automatically destroyed, but we recommend verifying this manually to ensure you aren’t unexpectedly billed.

Troubleshooting

In some cases, GitLab may report that the runner is unreachable and as a result perform no actions, including deploying new runners. You can troubleshoot this by stopping GitLab Runner, then starting it again in debug mode:

gitlab-runner stop
gitlab-runner --debug start

The output should throw an error, which will be helpful in determining which configuration is causing the issue.

If your configuration creates too many machines, and you wish to remove them all at the same time, you can run this command to destroy them all:

Conclusion

You've successfully set up an automated CI/CD pipeline using GitLab Runner and Docker. From here, you could configure higher levels of caching with Docker Registry to optimize performance or explore the use of tagging code builds to specific GitLab code runners.