I spent a substantial portion of Thanksgiving break working on Openaddresses Machine and publishing new data to data.openaddresses.io. Introducing any kind of reliable automation to a process like this is going to be a bumpy ride.

Ever since I made an automatic publishing infrastructure for highlight.js releases… there wasn't a single time when it really worked automatically as planned!

There was always something: directory structure changes that require updates to the automation tool itself, botched release tagging, out of date dependencies on the server, our CDN partners having their own bugs with automatic updates, etc. … You can't really automate anything. You just shift maintenance from your thing to your automation tools. But! It still makes sense because by introducing automation you can do more complex things at the next level and keep maintenance essentially constant.

So that’s where the non-daylight hours of my holiday weekend went: shifting maintenance from the thing to the automation tool so OA can do more complex things. Previously, I was running the OA data process using a multi-step process:

This error happens because { } has two different meanings in Ruby: Hash value expressions and method blocks. If a procedure is called in poetry mode (no parens) then there is an ambiguity if the parser encounters a { after a method name.

Boto and Amazon Web Services are blessedly stable; I was able to copy-paste code from three years ago and have it just work to guess a reasonable EC2 spot request bid and start up a server instance. One interaction I introduced was the result of finding a runaway 8xlarge EC2 instance from earlier this month that I had forgotten about and continued to pay for (it adds up): the instance terminates itself when it’s done, the run script monitors the instance on a loop, and canceling the script with a KeyboardInterrupt at any point will immediately terminate the instance and cancel the reservation. Just because a computer is in the sky doesn’t mean I don’t want a convincing illusion of running it in my own terminal.