Integration Testing With Docker Compose

Integration testing is often a difficult venture, especially when it comes to distributed systems. Even if you’re building a monolithic app you probably need to spin up a database to do integration testing. It’s also the kind of thing that’s simple to do early on, but gets exponentially harder as the codebase expands. Thankfully Docker Compose gives us the ability to do integration testing in any environment that runs docker.

Getting Started

Let’s say you’re staring with a monolithic setup, you’ve got one server and one database. Maybe you build the app server and database from source like it’s 1999, or maybe you use brew install to get all of your dependencies resolved. But at the end of the day your system looks like this.

The endpoint you’d like to test is /create and all it should do is store a some data in the database. Seems simple enough. So you write a bash script that CURLs and endpoint, and then queries the database (exit 0 for OK, exit 1 for FAIL). It’s easy AND most importantly it works.

This eliminates the last hidden dependency (existing database data), but also introduces a pretty nasty side effect. It’s only a side effect because the local development database is shared with the test database. So every time you run your integration test, you lose all of your development data 😭. This may seem obvious, but in practice this setup still exists. But it doesn’t have to be this way. From here on out, I’ll walk through an example built on top of Docker Compose that addresses all of the issues listed above. For this example I’ll use Node for the app framework and RethinkDB for the database, but there’s no reason why you couldn’t choose another stack.

Devise A Strategy

Let’s take a page from Martin Fowler’s microservice testing playbook for integration testing. We’re going to spin up a container outside of the system under test, have the container run some tests, and then check the exit code of the testing container’s run command.

For clarity I’d like to point out the file structure since we’re going to have multiple Dockerfile in the same project.

Ephemeral Database

Sometimes it’s nice to lose all your data, and when you’re running tests it’s essential. It’s really easy to accomplish this with Docker compose by spinning up your database without a mounted volume for data. This means that when you destroy your container, the data goes along with it. It also means that if you don’t destroy your container, you can exec into it and run queries against the database to debug. Here’s an example Docker Compose file that would just spin up an ephemeral database (RethinkDB).

integration-test/docker-compose.yml

version: '2'
services:
rethinkdb:
image: rethinkdb
expose:
- "28015"

Keep this concept in mind, because we’re going to use it soon.

Application Container

The next step is to containerize the application you’d like to test. It needs to build/run the application, link to the database and expose a port to be used for testing.

At this point you could sanity check the services with docker-compose up and go to http://localhost:8080 (so long as you had a server and routes wired up).

Integration Test Container

Now we’ve got our database and application, let’s build the testing container. This container needs to POST against the /create endpoint on my-service and inspect the database for changes. To accomplish this I used tape and request-promise to inspect the endpoint.

integration-test/index.js

importtestfrom'tape';importrequestPromisefrom'request-promise';constbefore=test;constafter=test;constbeforeEach=()=>{/*test setup*/};constafterEach=()=>{/*test cleanup*/};before('before',(t)=>{/*one time setup*/});test('POST /create',(t)=>{beforeEach().then(()=>(requestPromise({method:'POST',// yes! we can use the service name in the docker-compose.yml fileuri:'http://my-service:8080/create',body:{thing:'this thing',},}))).then((response)=>{// inspect the responset.equal(response.statusCode,200,'statusCode: 200');}).then(()=>(// inspect the databaserethinkdb.table('table_name').filter({thing:'this thing',}).count().run(connection).then((value)=>{t.equal(value,1,'have data');}))).catch((error)=>t.fail(error)).then(()=>afterEach()).then(()=>t.end());});after('after',(t)=>{/*one time setup*/});

So here’s the cool part, when you run docker-compose up a few things happen

my-service and integration-tester containers are built

my-service, integration-tester and rethinkdb are linked and ran

integration-tester runs all tests until it stops

after integration-tester stops, docker-compose spins down all containers

This is exactly what we need to run integration testing in CI. We still haven’t inspected the exit code of the integration-tester container, but I’ll get to that soon.

Bringing It All Together

With all of the automation in place we need to tie everything together and do some cleanup after the test finishes. To accomplish this we can use Docker wait to block the script and retrieve the exit code of the test. We’ll use that code to output a message (PASS/FAIL) and exit the master script with the same exit code. This is useful because most (if not all) CI environments use an exit code to determine if the tests passed or failed. We’ll also grab the test container logs and print them out to provide context for when things fail. Here’s an (extremely verbose) script that does everything we need to run our integration tests locally or in CI.

Use This Now (Yeoman Generator)

It brings together Docker, Node (ES6) and Travis CI. Please let me know if you find any weirdness, I love pull requests!

Conclusion

This approach has been working well in practice and I’ve been using it to do integration testing for a handful of microservices. Any time I had a failure in CI, sure enough the same bug would occur locally. The biggest issue I ran into was tests failing because the application wasn’t fully up. To fix this I implemented a /health api endpoint on the app and added a retry inside of the before block of the test. Since I fixed that issue I’ve had no other weirdness and have been using this to run integration tests in CI. This has been really useful and has caught some real bugs that would have probably surfaced during deployment, I hope you find it useful too!