Changing redis maxmemory gradually

Today we’ve been busy migrating some AWS instances at work due to upcoming
maintenance events in AWS. One of the instance families we used a lot when
building QuizUp is the m2 instance family and almost all of our m2 instances
will require restarts in the next few days.

On some of these we are running large redis instances with tens of GB of data
and restarting them is not pain-free, even if we would use the persistence
features of redis, which work by periodically dumping the entire dataset into
a dump.rdb file. If the keyspace is tens of GB, the process of saving, and
reading the data is quite slow. Even with a disk subsystem capable of 100MB/s
sustained reads and writes, saving and reading will take around 500 seconds each.

In the mean time you probably do not want to serve requests. While saving
because you’ll write changes that won’t be reflected in the dump.rdb file, so
you lose data on restart, and while reading the data redis does not allow writes
(at least by default) and even serving reads can be dangerous if your application
makes assumptions based on the availability of keys.

In order to deal nicely with this scenario we basically slave our redis instances
with newer instance types and then mostly seamlessly failover to them (I can
write another blog post on that later). Slaving however is not problem-free.

Until redis 2.8.18 the only way to slave is for the master to start by making a
dump.rdb file, which when complete gets streamed to the slave, which saves it
to disk, then reads it up from disk, and all writes/changes happening on the
master are buffered in the mean-time. This causes a series of problems (more
here)
which I won’t expand on here, but let’s suffice to say that the smaller the dataset
is, the easier time you will have making a slave of it.

If your dataset is mostly volatile, meaning that it’s nice to have the data there
but not crucial, lowering the maxmemory down to force redis to evict keys is a
sound strategy to improve your life as a slavemaster. Today I took a crappy script
I had which does just that and packaged it a little more nicely and it’s on GitHub:
steinnes/redis-memslider.