The lowest hanging fruit however is linting, which is where foodcritic comes in. Foodcritic parses your cookbook, and warns you about many common errors.

Food critic will yell at you about all sorts of things – if you are accessing node attributes inconsistently, if you’re passing valid ruby as a not_if string, if you’re using /tmp instead of file_cache_path, and many more. For a full list, see the github pages

Automatically running foodcritic on your public cookbooks is easy with Travis CI. Just add a .travis.yml to your repo that looks like this:

Last week Brian Cunnie posted a great writeup of what we’ve been working on for building OSX Lion workstations. Today, I’d like to introduce Apple Orchard – how we transform those chef recipes into ready to use OSX Lion images.

The story starts a few months ago, one morning after Standup while putting our dishes from breakfast away. Over the past few days we’d been discussing how our ops group would take chef recipes (generated for the most part by developers) and turn them into machine images that they could deploy on a moments notice. I approached Sean Beckett, our Director of Operations, and told him of my vision:

no manual steps

He looked at me like I was crazy, and he was obviously trying to figure out how to talk me down off the ledge. I told him how Jenkins could run a job after every checkin (a fact he was well aware of) and how all it had to do was…… He backed away slowly.

A few weeks later, Brian Cunnie had gotten past the Minimum Viable Image release marker in Tracker, and I told him of my vision:

no manual steps

He also looked at me like I was crazy, which he often does. The next week I had the afternoon to pair with him, and we got to work. We already had Jenkins building our recipes on a mac mini with Deep Freeze (software which allows you to reboot to a clean state), so we copied that Jenkins job at got to work. We got an iMac, and partitioned the disk into two.

We boot the image to the persistent side, and image the dynamic side with a “mostly pristine” image that has X Code preinstalled, and has SSH turned on. We then set the machine to boot from that partition, and reboot.

Reboot the machine to the persistent partition, and wait for it to come up. This isn’t part of Step 2 because we want to leave the machine in the dirty state for troubleshooting if the chef run failed, and only trigger this if it succeeds.

Put a script in place that will automatically rename the machine when it first boots, take an image of the partition using diskutils, scan it for restore, and move it over to our Deploy Studio server, and create a symlink so the ‘lion HEAD’ build points to the newly generated image.

That’s it. We occasionally promote a ‘lion HEAD’ build to ‘lion STABLE’, so that we’ve always got an image on hand that we’re confident in. But the overhead of cutting a new image is now simply changing a symlink.

There are a lot of moving parts, and sometimes it breaks. With time, it’s become more reliable, but still has a lot of external dependencies. We’ve recently been trying out a strategy of pre-populating the chef and homebrew caches, which seem to be helping. Another caveat we’ve run into is that so far with Lion we’ve been unable to produce a universal image that will boot both MacBook Airs and iMacs, but we hope this may have changed with the latest 10.7.2 update.

Many thanks to Brian Cunnie – while I was the reason for this madness, he’s done most of the heavy lifting with my occasional help.

I spent a large part of this week at Velocity and Devopsdays. I met a lot of great people and heard way more interesting things than I could absorb. Here’s a collection of things (in no particular order) I found interesting.

JSFiddle is a gist-list site for sharing HTML/CSS/JS snippets. Looks to be a great way to share useful examples and problems. jQuery uses it for bug reports.

Dyn’s DynECT seemed to be the standard answer for how to distribute load to different datacenters for load balancing and geo-targetting. There is no standard answer to defeating the CAP theorem.

There’s a huge movement to rebuild Splunk in various open source projects. Logstash is an open source project which is piecing together various open source components to get something similar, but is doing index time extraction instead of search time extraction and hasn’t gotten to graphing or analysis yet. Etsy is using Graphite extensively for metrics collection and say that developers prefer it to Splunk. Various other companies are building app-level only metrics collection/reporting solutions. The consensus in the devops community is that Splunk’s pricing does work for the web. As a fan of Splunk, I hope they can figure this out as it’s still better than anything I’ve seen.

Etsy’s work with metrics collection is interesting – they’ve written statsD for aggregating things before sending to graphite and Logster for tailing logs and making graphite events out of them.

Joshua Timberman of Opscode has some great slides on how to write a Chef cookbook and I wish I’d gotten to see the talk at Devopsdays. I tend to think they error on the side of premature extraction for reusability, but I see their point of view. I’ll definitely be using the remote_file resource and the file_cache_path.

2011 is shaping up to be the year of the Zookeeper alternatives. If you’re not familiar, Zookeeper is a reimplementation of Google’s Chubby – a highly available and reliable system designed to be the system of record for where services are currently available. Netflix has their own internal solution. Heroku has Doozer, and Noah is the new kid on the block with a simple rest interface and http callbacks(aka: webhooks). Opscode is also trying to answer this need by querying the Chef server at runtime for the identities of other hosts on the network. As environments and servers becoming ephemeral, telling every server about every server gets to be a tedious liability. It’ll be interesting to see where this goes.

Run Deck (aside from having a great logo) seems to be an interesting way to give users limitable, auditable, and repeatable access to server shells. It’s a web interface that lets you execute commands on various configurable subsets of your infrastructure. It’s got plugins so it can grab the current instances out of EC2, and a jenkins plugin so a Jenkins build can trigger a job in Run Deck. I’ll definitely be checking it out when I get a chance.

Pipe Viewer – (pv) Looks awesome. You can put it in a series of pipes in a shell, and it’ll print out a nice status bar about the amount of data passing through the pipe. ‘brew install pv’

Opscode has a “fat” installer of chef coming out. It goes in /opt/opscode and includes all the dependencies in the package. This should answer a common complaint about having to install ruby to install ruby. (Yo, dawg, I heard you like Ruby)

CSS Lint – Nicole Sullivan and Nicholas Zakas teamed up and introduced CSSLint at Velocity, which looks like it’ll a good addition to a lot of test suites. Nicole also mentioned some cool pure CSS buttons that work on dark and light backgrounds. Lea Verou’s pure css backgrounds were mentioned, which went around the office a while ago but are worth a link. It was also news to me that CSS selectors are run left to right by a browser – “ul.foo span.special *” will match all tags on the page, then narrow it down.

Blue/Green Deployment is all the rage the days, and for good reason. Netflix and Amazon are both using it. You’ll have to decide how it fits with your data storage layer, but it’s worked well for me in the past and I’m glad to see it gaining popularity.