Most of our code is based on Ruby on Rails and we are big fans of Test Driven Development. We deploy multiple times a day and that is why we want to spend as little time as possible on manual QA. Our test suite contains more than 14.000 unit, functional and integration tests with a code coverage of 98%. The build process of our core application has various runtime and infrastructure dependencies including:

A specific ruby version

Operating system packages(e.g. imagemagick, libxml, libmysql, etc.)

Gem dependencies(~150 gems defined in our Gemfile)

MySQL

ElasticSearch

Redis

Managing all these dependencies for our development and CI environments can be very complicated. This is why I want to outline how we use Docker Compose to solve this problem for us.

# next two steps will be cached unless Gemfile or Gemfile.lock changes.

# -j $(nproc) runs bundler in parallel with the amount of CPUs processes

COPY Gemfile Gemfile.lock/app/

RUN bundle install-j$(nproc)

We base our image on the default ruby image in the exact version we also run on production and then first install all required system packages. Parts of our frontend uses webpack and so we also add the repositories for nodejs and yarn.

We only add the necessary files to install our rubygem and npm dependencies(Gemfile, Gemfile.lock, package.json and yarn.lock) to the image. Unless any of these files change, docker can cache all steps resulting in almost no overhead.

Instead of also adding the actual application code to the image we just mount the directory with the code when running the tests.

All infrastructure services for our development environment and our integration tests (MySQL, ElasticSearch and Redis) have health checks configured and so the spring, test, dev and server services are only started when all of them are healthy.

As all services are placed in a dedicated docker network they can be accessed just by their name (e.g. DATABASE_URL=”mysql2://mysql/phraseapp_test?local_infile=true”).

There are dedicated services for the infrastructure components for the development and test environments. The spring and server services use a little script (validate-migrated.sh) to determine if migrations were already executed. If yes, we run possibly pending migrations rails db:migrate, otherwise we execute the more efficient rails db:setup.

Running tests

As we are dealing with a completely empty database we first need to load the database schema(rails db:setup ) before we can execute our tests(rails spec).

We use xvfb-run as a wrapper because parts of our integration tests use capybara webkit and therefore need to run inside a XServer.

Docker Compose reuses dependent services so we need to delete all of them after we are done with docker-compose down. This can cause problems when running multiple tests in parallel but we can use the–-project-name flag of docker-compose with e.g. a randomly generated name to fully isolate all test runs.

To make things simpler we created a wrapper script which automatically runs tests isolated and also cleans up afterwards:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

#!/bin/bash

set-e

# random name with timestamp as prefix to give a cleanup script more information

Running tests as developer

Because the setup and teardown part introduces quite some latency, this approach is not really practical for test driven development. This problem can be solved with the spring service which we also configured in the docker-compose.yml.

As a developer, I can start a long running spring process with all infrastructure dependencies inside a docker container

to execute selected tests with a much lower latency. Similar to the wrapper script above we can create a shell alias to make things simpler:

1

2

alias dc-spec='docker-compose exec spring xvfb-run spring rspec'

dp-spec spec/models/user_spec.rb

Running the development server

Running the development server is as simple as executing

1

docker-compose up server

By exposing port 3000 of the container we can access the server running inside docker at http://localhost:3000.

Conclusion

Using Docker Compose for the build and development process of a Ruby on Rails application has many advantages. The actual dependencies for development and CI environments can be boiled down to docker and docker-compose. Tools such as rbenv or rvm are no longer needed.

This approach also avoids typical dependency issues when installing gems which need to be compiled such as nokogiri or capybara webkit. Some of our developers use OSX, others different Linux distributions and some even Windows. With Docker Compose we now no longer need to maintain separate documents for how to setup the development environment. Instead of executing manual steps this process is now also fully automated.

As the docker-compose.yml is checked into our code base we can test our application against newer versions of Ruby, MySQL, ElasticSearch or Redis by just updating the specific versions in that file. There is no need to install any of these new dependencies in either the development environment or on our CI servers. Also, this makes sure that all environments use the very same well-defined versions of all components.

Outlook

Our full test suite currently takes almost an hour to run sequentially, so our developers never run the whole suite locally. Instead, we frequently push our local changes to a remote branch on GitHub and our CI infrastructure automatically builds all these pushes. To reduce these feedback cycles it is necessary for us to run our tests in parallel and on multiple CI servers.

In one of our next articles, we will dive deeper in how we do this based on the setup presented above.