Database Scalability Patterns: Sharding for Massive Growth

These days every database supports some form of replicated read slaves to enable horizontal scaling for read operations. That’s good, but what happens when you’re read traffic doesn’t spread uniformly across all nodes? What happens when you’re ability to write data and push it out amongst these nodes outstrips outstrips the capabilities of the master node? In these cases people turn to sharding, but how best to accomplish that isn’t quite the simple solution many people would like it to be.

In this talk we’ll focus on how to solve this problem; sharding data across multiple nodes to handle both read and write traffic, even in cases where that traffic might not be uniform. We’ll also cover how to keep things running; schema changes, index builds, and failover scenarios will all be covered, discussing the trade-offs and constraints that various database will impose upon you.

We’ll also discuss the other half of the equation; how this affects the applications that need to interface with the database. Sharded architectures add complexity to any environment, and navigating those waters can be tricky. We’ll cover some of the most common hurdles we’ve run across, and show how to stretch the patterns based on your needs.

Robert Treat

OmniTI

With more than fifteen years of experience building database backed, internet based systems at multiple fortune 500 companies, Robert is now CEO of OmniTI, a technical services firm focused on providing web application development and infrastructure management at scale. Author and speaker at conferences worldwide, Robert is a recognized expert within the industry on topics such as Open Source, databases, and managing operations at scale. He occasionally blogs at http://xzilla.net.