I'm a web developer in Norfolk. This is my blog...

9th November 2014 5:13 pm

Building a URL Shortener With Node.js and Redis

The NoSQL movement is an exciting one for web developers. While relational databases such as MySQL are applicable to solving a wide range of problems, they aren’t the best solution for every problem. Sometimes you may find yourself dealing with a problem where an alternative data store may make more sense.

Redis is one of the data stores that have appeared as part of this movement, and is arguably one of the more generally useful ones. Since it solves different problems to a relational database, it’s not generally useful as an alternative to them - instead it is often used alongside them.

What is Redis?

Redis is described as follows on the website:

“Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs”.

In other words, its core functionality is that it allows you to store a value by a key, and later retrieve that data using the key. It also allows you to set an optional expiry time for that key-value pair. It’s quite similar to Memcached in that respect, and indeed one obvious use case for Redis is as an alternative to Memcached. However, it offers a number of additional benefits - for one thing, it supports more data types, and for another, it allows you to persist your data to disk (unlike Memcached, which only retains the data in memory, meaning it’s lost on restart or if Memcached crashes). The latter means that for some very simple web applications, Redis can be used as your sole data store.

In this tutorial, we’ll build a simple URL shortener, using Redis as the sole data store. A URL shortener only really requires two fields:

A string to identify the correct URL

The URL

That makes Redis a good fit for this use case since all we need to do is generate an ID for each URL, then when a link is followed, look up the URL for that key, and redirect the user to it. As long as this is all our application needs to do, we can quite happily use Redis for this rather than a relational database, and it will be significantly faster than a relational database would be for this use case.

Getting started

We’re more interested in the fundamentals of using Redis in our application than a specific language here. As JavaScript is pretty much required to be a web developer, I think it’s a fairly safe bet to use Node.js rather than PHP or Python, since that way, even if your only experience of JavaScript is client-side, you shouldn’t have too much trouble following along.

You’ll need to have Node.js installed, and I’ll leave the details of installing this to you. You’ll also need the Grunt CLI - install this globally as follows:

$ sudo npm install -g grunt-cli

Finally, you’ll want to have Redis itself installed. You might also want to install hiredis, which is a faster Redis client that gets used automatically where available.

Now, let’s create our package.json file:

$ npm init

You’ll see a number of questions. Your generated package.json file should look something like this:

Let’s go through this code. First we confirm that linting tools should treat this as a Node app, and use strict mode (I recommend always using strict mode in JavaScript).

Then we declare our variables and import the required modules. Note here that we set the port to 5000, but can also set it based on the PORT environment variable, which is used by Heroku. We also define a base URL, which again can be overriden from an environment variable when hosted on Heroku.

We then set up our connection to our Redis instance. When we push the code up to Heroku, we’ll use the Redis To Go addon, so we check for an environment variable containing the Redis URL. If it’s set, we use that to connect. Otherwise, we just connect as normal.

We then set up templating using Jade, and define the folder containing our views, and store the base URL within the app. Then we set up bodyParser so that Express can handle POST data.

Next, we define our index route to just render the index.jade file. Finally, we set up our static folder and set the app to listen on the correct port.

Note that Istanbul will have generated a nice HTML coverage report, which will be at coverage/index.html, but this won’t show 100% test coverage due to the Heroku-specific Redis section. To fix this, amend that section as follows:

This route is fairly simple. It handles POST requests to the index route, and first of all it gets the URL from the POST request. Then it randomly generates a hash to use as the key.

The next part is where we see Redis in action. We create a new key-value pair, with the key set to the newly generated ID, and the value set to the URL. Once Redis confirms that has been done, the callback is fired, which renders the output.jade view with the ID and base URL passed through, so that we can see our shortened URL.

Our final task is to implement the short URL handling. We want to check to see if a short URL exists. If it does, we redirect the user to the destination. If it doesn’t, we raise a 404 error. For that we need two more tests. Here they are:

The first test creates a URL for testing purposes. It then navigates to that URL. Note that we set followRedirect to true - this is because, by default, request will follow any redirect, so we need to prevent it from doing so to ensure that the headers to redirect the user are set correctly.

Once the response has been received, we then check that the status code is 301 (Moved Permanently), and that the Location header is set to the correct destination. When a real browser visits this page, it will be redirected accordingly.

The second test tries to fetch a non-existent URL, and checks that the status code is 404, and the response contains the words Link not found.

We accept the ID as a parameter in the URL. We trim off any whitespace around it, and then we query Redis for a URL with that ID. If we find one, we set the status code to 301, and the location to the URL, and send the response. Otherwise, we set the status to 404 and render the error view.

Excellent! Our URL shortener is now complete. From here, deploying it to Heroku is straightforward - you’ll need to install the Redis to Go addon, and refer to Heroku’s documentation on deploying Node.js applications for more details.

Wrapping up

I hope you’ve enjoyed this brief introduction to Redis, and that it’s opened your eyes to at least one of the alternatives out there to a relational database. I’ll hopefully be able to follow this up with examples of some other problems Redis is ideal for solving.