Debugging Travis builds

Important Update December 19, 2013 — Travis infrastructure has changed since this post was written. This post remains unchanged for posterity; however, be aware that this setup no longer represents the current state of Travis.

Yeti uses Travis CI to the max by spawning lots of PhantomJS processes that are tested asynchronously. When tests start failing in Travis, but not anywhere else, debugging can be infuriating.

Prepare the jvm box

Start the jvm box

Create a working directory to hold the Vagrantfile for the travis-jvm box.

mkdir boxes
cd boxes

Create the Vagrantfile and get the box ready to use:

vagrant init travis-jvm

The username for the VM is travis, so add this line to your Vagrantfile’s do-block:

config.ssh.username = "travis"

Start the VM:

vagrant up

Things should work nicely. (If not, read on.)

You can access the VM with this command:

vagrant ssh

You can now follow the steps of your Travis build and debug your problem.

If you have problems running vagrant ssh, you can try using a username and password for logging in instead with this command:

vagrant ssh -p -- -l travis

The password is travis.

More fancy debugging

The VM uses a NAT-only network. You may not be able to access it from your computer. For the problems I debug, I need to be able to access the the Node web server inside the VM from a browser on OS X.

This means you should setup a host-only network interface, in addition to the NAT-only interface.

Add this line to your Vagrantfile’s do-block:

config.vm.network :hostonly, "192.168.89.10"

This will setup a host-only network and give the IP address 192.168.89.10 to the VM. (If you are on a network that uses 192.168.89.xxx already, change the IP subnet to something else, because it cannot conflict with another subnet your host computer is using.)

Do not debug on travis-ci.org

When you have a Travis-only problem, try creating a local environment as described above first. Don’t try to use the hosted Travis service to debug complicated failures. I have made this mistake many times:

Noticed my build is failing in Travis.

Make sure my own environments and clean installs do not fail. (The problem is sometimes here!)

Create a branch named fix-travis-foo or something.

Start making commits and pushes to this temporary branch attempting to fix the problem.

Wait for the build to finish. If the build failed, repeat step 4.

First of all, this takes a long time. Travis isn’t terribly slow for CI purposes, but when you’re trying to actively debug something, the last thing you need is having your app tests go from 15 seconds to 5 minutes.

Most of all, my problems are usually with PhantomJS. When I can fully control the VM, I can add remotely debug the headless browser on OS X. I can also capture packets creatively with tcpdump(8). Using those tools, I found some pretty heinous race conditions in the Yeti test suite that would have been nearly impossible to find by trial-and-error.

Published by

Reid Burke

Hi, I'm @reid. I work on developer tools at Yahoo. I serve at Awakening Church with a focus on technology. I also enjoy riding motorcycles.