Can you dig it?

HipChat's 1 Billion Messages Served

Jan 6th, 2014

There’s an instructive post on the HipChat blog regarding how their product’s message count has risen from 110 million to over a billion since they were acquired by Atlassian. Central to their scaling strategy to meet data growth was their migration from Lucene to Elasticsearch.

add more nodes to our cluster when we need more capacity, so we can handle extra load while concurrently serving requests. Moreover, the ability to have our shards replicated across the cluster means if we ever lose an instance, we can still continue serving requests, reducing the amount of time HipChat Search is offline.

Sharding in Redis isn’t supported out of the box and requires an application to determine how to partition data across shards. I wonder how HipChat data is partitioned and if their strategy evenly smears data across their shards – like with any sharing strategy, once you’ve sharded on a particular key, it’s extremely difficult to re-shard.