Thursday, September 16, 2010

Earlier this week, I participated in a talk at Hunch on Scaling Backend Infrastructure with Tom Pinckney (Hunch), Kiril Sheynkman (Thansys), and Jeff Hammerbacher (Cloudera).

How to scale inside the request loop

The majority of web applications have a user sitting behind a web browser who lands on a site, clicks on a button and expects something to happen - quickly. As more users visit your site, they compete for scarce resources - CPU cycles, RAM, hard disk access, and bandwidth. Your goal is to add more resources to support more users - here's how.

Build a system that can be scaled.

Make sure your hardware is clone-able.

If you lose one machine, you need to be able to build a new machine that exactly matches that old machine from OS to configuration.

Know how long it takes each component in your system to respond. From MySql explain plans to queue depth, to each webservice call, to end-user experience, know how long each component of a single request takes.

Most application monitoring and measurement requires some bit of custom coding.

Analyze real-world scenarios to design performance testing. Look at what users actually do on the site to develop your performance test plans. Identify the most common paths or the most frequently accessed pages.

Design performance tests with product/user experience folks to make sure how they expect users to use your site is captured.

Design your tests to validate horizontal scalability. Run your tests with one of everything. Add machines. What happens to your performance?