August 25, 2005

So, it's true, they really do deploy (up to) every half an hour. Pretty cool; In answer to the two questions from the previous entry:

Why do they do it?

Because they can. Because, for them at least, 'Release early, Release often' doesn't become any less effective, for any value of often. The smaller and quicker the releases, the less chance of regression, the faster features get to users, and the sooner feedback comes back to the team. Basically, they release pretty much every feature and bug-fix as soon as it's complete – they don't really bother with 'batching' releases like we do.

How do they do it?

There are a number of tricks:

– A true on-click build and deploy. It looks like this.

For most types of changes, the implementation is nothing more than a simple rsync from CVS HEAD onto the server farm. Sometimes an apachectl graceful, a squid/memcached flush or a restart of the various daemons is also required, but it's a zero-downtime thing even under peak loads.

– A similar on-click rollback to any previous version if it all goes wrong.

– A good separation of layers in the application to minimise liklihood of collateral damage from a change.

– A small team with pretty complete knowledge of the code that they're updating. (2 java programmers, 2 PHPers, a designer and a front-end guy)

– A component-based application infrastructure with clear interfaces between components. Most application logic is coded in PHP, with cacheing provided via Squid and memcached, and key long-running daemon processes in java. Individual components can be redeployed with comparatively little risk to other components, so long as interfaces aren't modified.

Surprisingly, there's not much weight put on automated testing. There are automated test suites for some key black-box components (the email parser for instance) and a few functional tess (written using perl's www:mechanize) but mostly they're reliant on developers testing on the staging server (though they now also have the services of yahoo's 'surfers' user-testing department).

Well. Quite frankly I'm jealous. Not that I don't think we could do it too, but it would be a lot of work to get there from where we are now. ISTM, also, that the benefits of of this set-up are a bit like doing XP; until you actually get everything working together you don't see much benefit, but once you do, suddenly it changes all the rules.