Just me being me. Continuously.

Monthly Archives: February 2015

We run a lot of background tasks where we don’t necessarily care where the code actually runs: it needs to have Internet access and access to a couple of S3 buckets, and that’s about it. Some of the jobs are appropriate for Hadoop and Spark, which we crank out through AWS ElasticMapReduce, but some is more nuanced.

etcd Backed

To begin with, we back everything with etcd. We use this CloudFormation to bootstrap an etcd cluster according to the CoreOS team’s recommendation for production workloads. We only run etcd on these machines, everyone else just looks to them for configuration data, and these machines do not run any other services.

Operations Tier

The operations tier of our stack provides essential services to the cluster: a Docker registry, Shipyard, and the Vulcand proxy. CloudFormation is used to load this cluster, after which we load fleet units to provide those services.

Vulcand’s documentation is lackluster, to say the least. While the docs give examples of how to set up your front and back ends with etcdctl, I’ve found that for some operations, you’re best using either the REST API or vctl. The Docker registry runs on its host on port 5000 and Shipyard on 8080.

I set up a hostname for the vulcand host, vulcan.int.derr.me, and then set up registry.int.derr.me and shipyard.into.derr.me as CNAMEs.

Incoming requests are handled by the matching front end. It fields the request to the attached backend, which is served by one or more servers. If we ever decide we’re pulling enough docker images that we should add another Docker server, we would simply add a second server.