Learning Redis

This post is part of my weekly tech learning series, where I take one hour each week to try out a piece of technology that I’d like to learn. Make sure to read to the end, where I have a screencast overview of the final application.

This week I decided to learn a bit more about Redis. I’ve used Redis in Chirk HR already but only as part of a non-critical service to see how it performs under load (I’d like to screencast this later). Today I wanted to use Redis as an actual data store but first, what is Redis?

What is Redis?

Redis is an open source key-value store. I think of it as a light nosql database with a lot of flexibility.

Today’s application

My idea for today was to build my running journal but this time as a CRUD style web app. I want to stay close to the metal so I can really see how Redis works so I picked my second favorite web framework: Sinatra.

Basically I wanted to mimic a Rails scaffold inside Sinatra using Redis as the data storage. I’m also using the redis gem because it is very similar to the actual Redis API. This means I’ll be implementing the data layer myself, without an ORM.

(Normally I’d use ActiveRecord or Datamapper for production applications instance but I’m more concerned with learning and exploration with this.)

Since I will be doing a lot of data layer work I’m also opting for heavy TDD with this app, using minitest and autotest.

Use TDD for the application behavior

After getting everything hooked up I decided to split the tests into two files:

test_redis_running.rb to test the RedisRunning Sinatra application at the integration layer. This would be the controller and view if you’re thinking about it as a MVC app.

test_run.rb to test the Run class, which is used as the data layer between the application and Redis.

I like to start with integration testing and write pending tests for each major workflow or feature. I came up with the following integration tests to get started with:

test_get_root_response

test_get_root_list_runs

test_get_root_add_form_present

test_post_new_with_valid_run

test_post_new_with_invalid_run

test_get_edit_response

test_get_edit_show_form

test_put_update_with_valid_update

test_put_update_with_invalid_update

test_delete_response

test_delete_should_delete_record

This is very similar to Rails scaffold except I decided to put the new form on the list of runs (:index) and to skip the details of a run (:show). With such a small data model, a list provides a perfect summary.

The first test I wrote was to check that getting the index was successful, which isn’t about Redis at all so I won’t bore you with the details of it.

The second test on the other hand (test_get_root_list_runs), is the important one (and as you’ll see later is all that I had time for). Here I wanted to do a few things:

Create a few runs and save them to Redis

Load the index page

Check that the runs were displayed

In order to implement this test I have to do a full cycle of saving and loading runs from Redis. This meant I’d need to build the data layer now.

Run accessors and attributes

Additionally, I’m going to need some way to uniquely identify each run. In Postgres I’d just use an id field as a primary key but from what I see, Redis doesn’t have primary keys. Only keys.

At first I thought of using the date as the primary key, but many runners will run multiple times per day so that won’t work. I could use a combination of the date and another field like distance or duration but there is also a chance those will conflict also (e.g. two runs per day, around a 1 mile track at a specific pace).

Redis doesn’t have primary keys so you might need to roll your own.

So I’ll need to be creating my own unique ids for each run. I’ve used UUIDs before and they seem like a good fit here.

So using attr_accessor and initialize I can easily fulfill these requirements.

# Way up top outside the Run class...require'securerandom'# In the Run class...def id
@id||= SecureRandom.uuid@idend

By using ||= I can make sure that if a Run already has an id then I won’t generate it again (existing record).

I also wrote a brute force test to make sure Ruby’s UUIDs were unique and that I wasn’t misreading the docs. It’s not really production quality but good for exploration of a new API:

def test_id_should_be_unique
# Lets create a bunch of runs and check their uniqueness.# TODO: not the best solution but time is limited
runs = 100.times.collect{ Run.new.id}
assert_equal 100, runs.uniq.length, "Duplicate ids found"end

Saving a run to Redis

Now that I have a data format in Ruby, I need to save it to Redis. Since Redis is a key value store, I decided to save my runs where the key is the id (uuid) and the data is a hash of strings. This way it would be easy to load the data back into a Ruby object.

Since the integration test uses the #save method, that’s what I need to implement first. After writing two quick tests I’m ready to figure out how to save to Redis:

Redis uses a #set method to save data. It takes the key and value of the data and will return an “OK” when it succeeds.

One thing that tripped me up for a minute was that if I just sent a Ruby hash to Redis it would get saved but as the string interpolation of the hash like

'{"f"=>"g"}'

This wouldn’t work because I’d need to use eval to turn that back into a Ruby hash if I wanted to load it into a Run object.

So I turned to my old standby, JSON. By calling #to_json on the Ruby hash before sending it to Redis, I can be sure that I can unserialize it on the way back out. This is what Redis stored when I encoded the hash above into JSON.

'{"f":"g"}'

That figured out, the #save method became easy. Just use #set to save the JSON using a run’s uuid as the key.

Why not use the native hash support in Redis?

Redis natively supports hashes with operations like hset, hget, and hkeys. I probably should have use them instead of JSON but I forget all about them until after I was done. Switching to a native Redis hash wouldn’t be difficult at all, because all of that logic is hidden inside two methods of the Run class. The Sinatra application itself doesn’t even know that Redis is used at all.

Loading a Ruby object out of Redis

The next step in the integration test is to load an object out of Redis and into Ruby, to be used by erb when creating the HTML page.

Mimicking ActiveRecord’s API again, I decided to name this method #find and only support finding a run by it’s uuid. The test is simple:

Notice how I’m using an actual run object and its attributes this time instead of a raw hash from Redis. This is what the Run data model class is providing me.

Thinking about the #find method, it needs to do two things:

Load the raw data from Redis, if it’s available

Convert the raw data into a Run object

Loading the data is quite easy. Redis uses the #get method to fetch a single object, all you have to do is give it the key. If it finds an object, it is returned. If Redis doesn’t find the object, you’ll get a nil back. This makes it simple to use an if statement for control flow between these two cases.

Since the raw data is JSON, I needed to feed it through a JSON parser to convert it to a Ruby hash. This isn’t all I needed though, I wanted a Run class so I can interact with it right away. Since my earlier work created an initialize method that could set attributes based on a hash, I could easily send the data there and get a Run object back out.

Where are we now?

Going back up the stack to the integration test, the data layer (Run) works for initialing, saving, and loading a record. That means all I have to do is to load all of the Runs from Redis and show them on the index page.

But, I’m out of time now.

I already have an idea of how I’d go about listing the Runs, using Redis’s keys method which returns all keys and then using mget to get multiple keys in one call. After that it’s just a delete action and the HTML.

Summary

Even though I didn’t get any user interface written, I created most of the data model and had some good tests to make sure the data was behaving the way I wanted. Finishing up the application would be straight-forward and would just take some more time than I have available.

Like I said in the introduction, I have Redis running in production for Chirk HR while I test how easy it is to keep running. As of this writing, the Redis server has been online and receiving data for over 16 days and 16 hours with no downtime (probably longer, my monitoring system lost its network connection for a bit).

Given this exploration and my production test I’ve gotten very comfortable with Redis and will be adding it to my developer toolbox for future apps. I think it’s great when you need a simple and stable key/value store.