Working with Redis in Go

26th February 2016

In this post I'm going to be looking at using Redis as a data persistence layer for a Go application. We'll start by explaining a few of the essential concepts, and then build a working web application which highlights some techniques for using Redis in a concurrency-safe way.

This post assumes a basic knowledge of Redis itself (and a working installation, if you want to follow along). If you haven't used Redis before, I highly recommend reading the Little Book of Redis by Karl Seguin or running through the Try Redis interactive tutorial.

Installing a driver

First up we need to install a Go driver (or client) for Redis. A list of available drivers is located at http://redis.io/clients#go.

Throughout this post we'll be using the Radix.v2 driver. It's well maintained, and I've found it's API clean and straightforward to use. If you're following along you'll need to go get it:

$ go get github.com/mediocregopher/radix.v2

Notice that the Radix.v2 package is broken up into 6 sub-packages (cluster, pool, pubsub, redis, sentinel and util). To begin with we'll only need the functionality in the redis package, so our import statements should look like:

import (
"github.com/mediocregopher/radix.v2/redis"
)

Getting started with Radix.v2 and Go

As an example, let's say that we have an online record shop and want to store information about the albums for sale in Redis.

There's many different ways we could model this data in Redis, but we'll keep things simple and store each album as a hash – with fields for title, artist, price and the number of 'likes' that it has. As the key for each album hash we'll use the pattern album:{id}, where id is a unique integer value.

So if we wanted to store a new album using the Redis CLI, we could execute a HMSET command along the lines of:

To do the same thing from a Go application, we need to combine a couple of functions from the Radix.v2 redis package.

The first is the Dial() function, which returns a new connection (or in Radix.v2 terms, client) to our Redis server.

The second is the client.Cmd() method, which sends a command to our Redis server across the connection. This always returns a pointer to a Resp object, which holds the reply from our command (or any error message if it didn't work).

In this example we're not really interested in the reply from Redis (all successful HMSET commands just reply with the string "OK") so we don't do anything with the *Resp object apart from checking it for any errors.

It's worth pointing out that, when we use these helper methods, the error they return could relate to one of two things: either the failed execution of the command (as stored in the Resp object's Err field), or the conversion of the reply data to the desired type (for example, we'd get an error if we tried to convert the reply "Jimi Hendrix" to a Float64). There's no way of knowing which kind of error it is unless we examine the error message.

If you run the code above you should get output which looks like:

$ go run main.go
Electric Ladyland by Jimi Hendrix: £4.95 [8 likes]

Let's now look at a more complete example, where we use the HGETALL command to retrieve all fields from an album hash in one go and store the information in a custom Album struct.

Using in a web application

One important thing to know about Radix.v2 is that the redis package (which we've used so far) is not safe for concurrent use.

If we want to access a single Redis server from multiple goroutines, as we would in a web application, we must use the pool package instead. This allows us to establish a pool of Redis connections and each time we want to use a connection we fetch it from the pool, execute our command on it, and return it too the pool.

We'll illustrate this in a simple web application, building on the online record store example we've already used. Our finished app will support 3 functions:

Method

Path

Function

GET

/album?id=1

Show details of a specific album (using the id provided in the query string)

POST

/like

Add a new like for a specific album (using the id provided in the request body)

GET

/popular

List the top 3 most liked albums in order

If you'd like to follow along, head on over into your Go workspace and create a basic application scaffold…

…And use the Redis CLI to add a few additional albums, along with a new likes sorted set. This sorted set will be used within the GET /popular route to help us quickly and efficiently retrieve the ids of albums with the most likes. Here's the commands to run:

We'll follow an MVC-ish pattern for our application and use the models/albums.go file for all our Redis-related logic.

In the models/albums.go file we'll use the init() function to establish a Redis connection pool on startup, and we'll repurpose the code we wrote earlier into a FindAlbum() function that we can use from our HTTP handlers.

Something worth elaborating on is the pool.New() function. In the above code we specify a pool size of 10, which simply limits the number of idle connections waiting in the pool to 10 at any one time. If all 10 connections are in use when an additional pool.Get() call is made a new connection will be created on the fly.

When you're only issuing one command on a connection, like in the FindAlbum() function above, it's possible to use the pool.Cmd() shortcut. This will automatically get a new connection from the pool, execute a given command, and then put the connection back in the pool.

Using transactions

When a user likes an album we need to issue two distinct commands: a HINCRBY to increment the likes field in the album hash, and a ZINCRBY to increment the relevant score in our likes sorted set.

This creates a problem. Ideally we would want both keys to be incremented at exactly the same time as a single atomic action. Having one key updated after the other opens up the potential for data races to occur.

The solution to this is to use Redis transactions, which let us run multiple commands together as an atomic group. To do this we use the MULTI command to start a transaction, followed by the commands (in our case a HINCRBY and ZINCRBY), and finally the EXEC command (which then executes our both our commands together as an atomic group).

Let's create a new IncrementLikes() function in the albums model which uses this technique.

File: models/albums.go

...
func IncrementLikes(id string) error {
conn, err := db.Get()
if err != nil {
return err
}
defer db.Put(conn)
// Before we do anything else, check that an album with the given id
// exists. The EXISTS command returns 1 if a specific key exists
// in the database, and 0 if it doesn't.
exists, err := conn.Cmd("EXISTS", "album:"+id).Int()
if err != nil {
return err
} else if exists == 0 {
return ErrNoAlbum
}
// Use the MULTI command to inform Redis that we are starting a new
// transaction.
err = conn.Cmd("MULTI").Err
if err != nil {
return err
}
// Increment the number of likes in the album hash by 1. Because it
// follows a MULTI command, this HINCRBY command is NOT executed but
// it is QUEUED as part of the transaction. We still need to check
// the reply's Err field at this point in case there was a problem
// queueing the command.
err = conn.Cmd("HINCRBY", "album:"+id, "likes", 1).Err
if err != nil {
return err
}
// And we do the same with the increment on our sorted set.
err = conn.Cmd("ZINCRBY", "likes", 1, id).Err
if err != nil {
return err
}
// Execute both commands in our transaction together as an atomic group.
// EXEC returns the replies from both commands as an array reply but,
// because we're not interested in either reply in this example, it
// suffices to simply check the reply's Err field for any errors.
err = conn.Cmd("EXEC").Err
if err != nil {
return err
}
return nil
}

We'll also update the main.go file to add an addLike() handler for the route:

Using the Watch command

OK, on to our final route: GET /popular. This route will display the details of the top 3 albums with the most likes, so to facilitate this we'll create a FindTopThree() function in the models/albums.go file. In this function we need to:

Use the ZREVRANGE command to fetch the 3 album ids with the highest score (i.e. most likes) from our likes sorted set.

Loop through the returned ids, using the HGETALL command to retrieve the details of each album and add them to a []*Album slice.

Again, it's possible to imagine a race condition occurring here. If a second client happens to like an album at the exact moment between our ZREVRANGE command and the HGETALLs for all 3 albums being completed, our user could end up being sent wrong or mis-ordered data.

The solution here is to use the Redis WATCH command in conjunction with a transaction. WATCH instructs Redis to monitor a specific key for any changes. If another client modifies our watched key between our WATCH instruction and our subsequent transaction's EXEC, the transaction will fail and return a nil reply. If no client changes the value before our EXEC, the transaction will complete as normal. We can execute our code in a loop until the transaction is successful.

One note about WATCH: a key will remain WATCHed until either we either EXEC (or DISCARD) our transaction, or we manually call UNWATCH on the key. So calling EXEC, as we do in the above example, is sufficient and the likes sorted set will be automatically UNWATCHed.

Making a request to the GET /popular route should now yield a response similar to: