The odyssey of a philomath of technology

Menu

Just for kicks, I wanted to try using an ECDSA key for ssh authentication. Unfortunately, the OpenSSH bundled with Mountain Lion (10.8) does not support ECDSA keys (nor can one even be generated with ssh-keygen.) The man pages for ssh-keygen and ssh-agent say they support ECDSA, but this is due to a naive man page generation assuming that since the OpenSSL library supports it, OpenSSH will too. Also, a PCI compliant OpenSSH isn’t bundled with OS X Lion (10.7) or older so this will also be useful for those users as well. Thankfully, Homebrew already has a recipe for installing an up-to-date OpenSSH so most of the work of upgrading is already done.

Now that PyPi is being accelerated by the Fastly caching network, pip/easy_install already are running faster. However, this can be taken a step further by setting up a simple caching proxy. By caching packages locally (on the machine or in your private network), you don’t have to keep hitting Fastly/PyPi to download them. This is especially useful if you are constantly running builds and/or tests: AKA continuous integration.

Phabricator is an awesome suite of tools open sourced by Facebook and now maintained by Phacility. At Disqus, it’s the central nexus of our engineering team. Since so much of an engineer’s day revolves around using the web interface, I was tasked with trying to optimize our local instance of it. The quickest win was installing and enabling APC per the Installation Guide. Next up, I opened the network tab of Chrome’s developer tools and found that PHP is handling the serving of static assets. Granted, phabricator does set very sane and liberal headers so that browsers will heavily cache all the assets, each browser still needs to obtain them first. To ease the pain of the first load, I setup Nginx to handle caching them as well. This way PHP only has to serve and/or generate assets once and something that’s far better at serving static content, can handle the heavy lifting from then on out.

In the upcoming 0.4 release of the nginx-push-stream-module, it will have support for the Nginx Gzip filter. Being able to gzip messages will free up bandwidth and decrease latency when under high load. However, the default deflate settings Nginx uses are not ideal for the high concurrency and small messages that are typically sent with the push-stream module. By default, Nginx may allocate up to a relatively large (264kb) chunk of memory for zlib upfront for every request that supports gzip. This adds up fast when there are thousands of concurrent connections to Nginx.

With the release of Cassandra 1.2, many new metrics were instrumented with Metrics with CASSANDRA-4009. However, getting those metrics into something like Graphite was still a polling process. Metrics does have Reporters that let Java Agents push metrics stored in the registry to various datastores (Graphite, Ganglia, etc.) Currently, this requires writing the agent code, compiling it and loading it into Cassandra. Soon there will be a way to just configure these reporters using metrics-reporters-config with CASSANDRA-4430. For now though, this simple agent will push metrics into Graphite while filtering out some noise.

Datastax has a blog post with a brief outline of how to enable the GraphiteReporter but it doesn’t go into much detail or release any code. This post augments it with the missing pieces.

We’ve been using Phabricator for just about a year here at Disqus. It was originally created at Facebook and open sourced in Spring 2011. To sum it up using their own words: “Phabricator is a open source collection of web applications which make it easier to write, review, and share source code.” The small team working on it at Phacility (the SaaS company behind Phabricator) is constantly improving it so it’s on a continuous release cycle.

Jenkins has been used for continuous integration testing here for much longer. I’m not exactly sure for how long since it was setup before I started in September 2011. David Cramer has always been pushing for an ideal continuous integration/deployment system (IE herehere) so part of my duties has been to improve what we have to achieve that goal (we’re hiring).

Currently, there isn’t a direct CI hook into Phabricator that is as deep as say Github+Travis. However, with a little script and an simple event listener for Arcanist, we can replicate most of that functionality.

Our lead operations engineer, Scott, put together a nice system called fpm-recipes using Git, GNU Make and FPM to keep track of how we build DEB packages of various things at Disqus. Instead of each ops engineer having their own way for building packages that are stored in various places (IE: shell history) we now have a centralized and standardized system. No more do we have to ask each other to update a package they maintain or curse ourselves for not saving the steps somewhere organized/accessible.

In no time I was able to get erlang-nox and zeromq recipes written (since they haven’t been updated in Ubuntu 10.04 LTS (Lucid Lynx) in ages). However, when I went back and tried to add their dependencies, things got a little hairy. GNU Make’s foreach function assumes lists “are whitespace-separated words”, so having something like DEPENDS := "libuuid1 (>= 2.16)" really doesn’t work as intended when passing it to foreach. So I wrote a function, quoted_map, that will map another function of a quoted list of strings. In fpm-recipes, it adds the -d and makes sure it’s quote (-d "libuuid1 (>= 2.16)") and adds to the FPM args list.

Per recommendation from a neckbeard friend, Aaron, I set out to try out Mutt as my email client. Since my email is hosted by Gmail, there’s a little extra configuration needed than just setting up an IMAP inbox. Also, since people actually send multimedia emails, I wrote a small patch for Mutt that detects it’s talking to a Gmail IMAP server and adds a couple custom headers to the message, one of which is the permalink to the email so it can be easily opened in a browser if need be. I’m sure I am one of the few that actually like Google Contacts, so I use Goobook for address completion. And no reason to go through all the trouble of setting up Mutt and not setup GPG for signing/encryption too. I am a fan of Ethan Schoonover’s Solarized color scheme, but I prefer a bit more contrast: I modified the Mutt colors Solarized Dark 16 colors for this preference.

I recently was setting up a couple ElasticSearch and RabbitMQ instances when I noticed RabbitMQ was still reporting an abysmally low fd limit in its log file at startup. I double checked my /etc/security/limits.conf and sure enough, limits were properly set to 64000. Yet for some reason it was still only seeing a max of 1024.

It turns out that in Ubuntu 10.04, /etc/pam.d/common-session{,-noninteractive} does not contain:
session required pam_limits.so