Building a Real-time SMS Voting App Part 1: Node.js & CouchDB

This is the first in a multi-part series of blog posts on building a real-time SMS and voice voting application using Node.js. In part one, we will create the Node.js application, set-up the CouchDB database and connect everything to Twilio so that we can process votes via SMS.

A few months ago, I was attending Startup Weekend GOV in Seattle where Twilio was a sponsor. I was there to support the event and help out teams that needed assistance with marketing validation, prototyping and pitching. I love helping out with Startup Weekend events, but sometimes I get a little antsy and wish I was working on something of my own. During this event, I happened to have some downtime so I decided to scratch my itch and start working on Votr, a SMS voting application built on Node.js.

Although I have been building web applications since 1999 and have used Java, PHP, Python and Ruby, this was my first experience using technologies like Node.js and CouchDB. In all past projects, I had used a RDBMS like MySQL, typically with an language-specific ORM. Having transitioned fairly painlessly between these technologies, I was shocked at how clueless I felt writing a Node.js app and working with a NoSQL database like CouchDB.

I thought it might be useful for other people who are making a similar transition to write about my experience coding Votr, the pitfalls I ran into and a few solutions that I discovered. In this first post I’ll simply walk through the basics of getting started: creating an app, setting-up the database and processing votes over SMS.

Create a Node App

First, you’ll need to have Node.js and NPM installed. For the remainder of this blog post, I’ll be rocking the terminal on Mac OS X, but these steps should be easy to follow on Windows or Linux. Here’s how you install Express and create an app:

Shell

1

2

sudo npm install express-g

express-Hvotr-part1 cdvotr-part1/npm install

At this point, you should have a starter node application. Let’s go ahead and deploy this to the cloud.

1) Sign-up for a Nodejitsu account

For the purpose of this tutorial I’m going to use Nodejitsu to deploy our Node app.

2) Install Jitsu, the command-line tool for Nodejitsu

3) Deploy the app

Shell

1

jitsu deploy

You’ll be asked to enter a subdomain for your application and the version of Node.js to use. The defaults should work fine, so feel free to hit enter. Once your deployment has finished, check to see that it’s running:

Set-up CouchDB

Setting-up a local CouchDB is easy, but for the purposes of this tutorial I’m going to show you how to use Cloudant’s hosted CouchDB. I evaluated a few different hosted solutions and experienced the fewest issues with Cloudant. I really came to love their web interface for CouchDB.

Before we dive in it, you might be wondering how CouchDB (and NoSQL in general) is different than something like MySQL. While that’s outside of the scope of this tutorial, below is a table that compares some of the core concepts from MySQL to CouchDB:

RDBMS

NoSQL

Database

CouchDB server

Table

Database

Row

Document

Column

things get tricky…

1) Sign-up for a Cloudant account

2) Create a database called “events”

3) Create a new document

4) Populate the document with some test data

Unlike a SQL database, there is no concept of a schema. I don’t have to define in advance what kind of attributes by documents are going to have. If you are starting twitch uncontrollably, don’t worry. There are other ways to enforce data integrity in a NoSQL database like validation functions, but a schema isn’t one of them.

In our app, our event documents will be structured like this:

Name: the name of the event where voting is taking place

Short Name: a URL-friendly version of this

State: the state of voting (on, off, etc)

Vote Options:an array of options for voting, where each option look like

Id: number to use for voting

Name: the name to display

Votes: the number of votes cast for this option

Numbers: an array of phone numbers that have voted for this option

Let’s go ahead and load a dummy event document into our database so we can get on with building our app.

JavaScript

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

{

"name":"Demo Voting",

"shortname":"demo",

"voteoptions":[{

"id":1,

"name":"foo",

"votes":0,

"numbers":[]

},

{

"id":2,

"name":"bar",

"votes":0,

"numbers":[]

},

{

"id":3,

"name":"baz",

"votes":0,

"numbers":[]

}],

"phonenumber":"change this later",

"state":"on"

}

5) Create a view to query events

Once our app is up and running, we’ll need to be able to query events. In the world of CouchDB, you use views to query documents. For anyone who has worked with Oracle databases, this will sound familiar. In Oracle, a view is simply a predefined SQL query that behaves like a read-only table. In CouchDB, a view is a map function that processes all of the documents in your database and created a hash of key/value pairs. These functions are stored inside of special documents called Design Documents. In our app, we’ll want to be able to query events based on their phone number and their short name. Let’s go ahead and create a design document and paste in the content of the Gist below.

1) Create utils.js

2) Edit package.json and add some dependencies

While we defined a few functions of our own above, but there’s nothing cool about reinventing the wheel. We’re going to make use of two NPM modules (cradle and twiliosig) so let’s go ahead and add those dependencies to our package manifest file.

4) Create events.js

This module is going to encapsulate access to our events database on CouchDB and provide some nice and simple methods for the rest of our app. This way, if we change our mind about CouchDB and decide to use a different datastore, we’ll just have one file to edit. Let’s define three methods:

findBy – looks up an event based on a key

hasVoted – checks to see if a person has already voted for a specific event

6) Edit app.js

7) Create views/forbidden.hjs

In the event that someone figures out the SMS Request URL of your voting application (because, I don’t know, it’s posted both here and on Github!) they might be tempted to try to generate phony requests and hijack the voting. Thankfully, this application checks X-Twilio-Signature and displays the following video to would-be intruders.

Wire Incoming SMS Messages to Our App

Ok, we’re almost done! The last thing we need to do is set-up a phone number to process incoming votes over SMS.

4) Deploy to Nodejitsu

5) Edit the event on Cloudant to reflect your new Twilio phone number

Let’s Vote!

At this point you can pull out your phone and start testing your voting app. Here’s a simple script I use:

Send a valid vote: text “1” (no quotes) to the voting number

Send an out of range vote: text “7”

Send an invalid vote: text “A”

Try to vote twice: text “1”

Below you’ll see the results of these text messages in Nodejitsu logs for your app:

Let’s double check our DB on Cloudant to verify the vote came in:

That concludes part one of building a real-time SMS voting application using Node.js. All of the source code for this application can be found on Github: https://github.com/crabasa/votr-part1.

You may be wondering where the “real-time” part was. In part two, we’ll build a visualization for this voting application using Highcharts JS and animate this visualization in real-time using Socket.io.

I’ve run into an issue that maybe you can help me with. I keep getting a 403 from /vote/sms. It reaches Twilio and Nodejitsu, but for some reason nodejitsu returns a 403. I thought this might be due to the incorrect Twilio auth keys, but I’ve double checked them and they’re correct. Any ideas?

Colin

I bypassed it using “config.disableTwilioSigCheck = true;” in config.js but I’ not sure why the Twilio auth keys are not working.

Great Tutorial. I have a few concerns about Cloudant for scaling however, as they charge for data and I/O. On a massive scale this cost would be enormous, and buying dedicated nodes is very expensive. Have you considered using you own server, and Futon, (CouchDB’s built in web GUI). I found it to be easier than Cloudant’s and it also lets you control replication, (I couldn’t seem to find where replication is triggered in Cloudant).

It looks like the “event” variable is null. Make sure that in your event document in CouchDB the value for “phone number” is in E.164 form (i.e. +15551234567). Shoot me a tweet @carterrabasa if you need any more help.