In this blog post I summarize the setup of a local Cassandra cluster with Docker on Mac OS X for chaos testing on a single machine. Chaos is generated by a coordinator written in Scala that randomly kills and restarts cluster nodes. I used boot2docker 1.6.2 and Mac OS X 10.9.5 but the following should also work with newer versions. It is assumed that the boot2docker VM is running:

Running the cluster

There are several Cassandra Docker images available. The image used here is that from the Docker Official Images project. For starting a three node Cassandra 2.1.6 cluster, a seed node is started first:

Accessing the nodes

The started containers are not directly accessible from Mac OS because they are running within the boot2docker VM. The goal however is to access these containers directly from Mac OS. This can be achieved with port mapping or custom routing.

Port mapping

With port mapping, container ports are mapped to boot2docker ports so that applications that are running on Mac OS can connect to the boot2docker VM. For example, when starting the seed node with

docker run --name cassandra-1 -d -p 9042:9042 cassandra:2.1.6

a connection to that node can be established with the boot2docker IP address:

Port mapping works well for tools like cqlsh but can be problematic when using the Datastax Java Driver, for example: nodes that are added to the cluster are advertised to the driver with their container IP addresses which cannot be accessed if the driver is running directly on Mac OS. In this case, a custom route can be added to the Mac OS routing tables that routes all traffic targeted at docker containers to the boot2docker VM, as shown in the next section.

Custom routing

Assuming that Docker containers have IP addresses 172.17.x.x, a custom route to the boot2docker VM can be added with:

Generating chaos

For randomly stopping and restarting nodes, the utilities project provides a coordinator application named ChaosCluster which can be started from sbt and configured with the parameters defined in reference.conf. Running ChaosCluster with default settings first starts a four node cluster:

The ChaosCluster application uses the scala.sys.process API for executing docker commands (see ChaosCommands trait for details). Later versions of the project will additionally provide utilities for running distributed Eventuate applications in Docker containers that are also subject to random failures.