Storing complex data structures in Redis

Jun 21, 2017

We use various data structures (linked lists, arrays, hashes, etc) in our applications. They are usually implemented in memory but sometimes we need persistence AND speed. This is where in memory DB like Redis can be very useful.

Redis has a number of powefull data types but what if we need something more complex? In this post I would like to go through commonly used data structures and see how they can be implemented using underlying Redis data types.

One of the advantages of these approaches is that we can restart some of the application processes or even shutdown parts of the system for maintenance. Data will be stored in Redis awaiting to be processed.

The gem uses mapped_hmset to store and hget to fetch data. It also uses OpenStruct to return an object to access attributes using user.email vs. user['email'].

Arrays

What if we have an array of email addresses emails = ['user1@email.com', 'user2@email.com', ...] that we need send messages to. Since this process can take a long time it would be nice to persist the data.

Using Lists

# save the recordsREDIS.lpush('array',emails)# data in Redis{"db":0,"key":"array","ttl":-1,"type":"list","value":["user2@email.com","user1@email.com",...]...}#classEmailSenderdefperform# check if there are still records using LLENwhileREDIS.llen('array')>0# fetch email addresses in batches of 10emails=REDIS.lrange('array',0,9)emails.eachdo|email|# send email code hereREDIS.lrem('array',1,email)# O(n) complexityendendendend

Using Sets

Alternatively we can persist our array in Redis Sets. Sets do not allow repeated members so that will ensure our email addresses are unique (which could be desirable or not). There are a few different ways we can fetch needed records from Redis.

We can use REDIS.smembers('array') which will return all records at once (but we might not want that). We will then use REDIS.srem(array, email) (which is O(n) complexity) to remove records after sending each one. But if our application crashes in the middle of sending we will still have unsent email addresses saved in Redis.

We can use combination of REDIS.srandmember('array', 10) to fetch emails in batches of 10. Then we loop through the batch, send the messages and REDIS.srem(array, email). srandmember is also O(n) complexity.

And we can use REDIS.spop(array) which will remove and return a random member with O(1) complexity but we will have to send emails one at a time. Usually the performance impact of making an outbound request to send email is greater than Redis operations so I would stay away from using spop. To scale this code we can make 10 spop iterations, store those emails in temp array and call email service provider API passing those addresses.

# save the recordsREDIS.sadd('array',emails)# data in Redis{"db":0,"key":"array","ttl":-1,"type":"set","value":["email1","email2",...]...}#classEmailSenderdefperform# check if there are still records using SCARDwhileREDIS.scard('array')>0# fetch and remove records using options above# send emailendendend

Stacks and Queues

Stacks and Queues can also be implemented with Redis Lists. Let’s imagine an API endpoint that receives messages http://localhost:3000/stack?my_param=foo