This post is the first of a three-part series showing how to build a complete javascript architecture using :

NodeJS : For the server side. We will use Socket.IO to manage long-terms real-time connection.

Google Chrome Extension : For the client side. WebSocket, Notification and Local Storage will be used.

MongoDB : To store the datas.

To illustrate our architecture we are going to create a Node application that will track tweets about the “What’s Next” event and broadcast them in real time to the clients. The client will be a Google Chrome browser extension and will use two features of the HTML5 specification to display the tweets broadcasted by the server. In the last post I will introduce the MongoDB database, we will use it to store tweets and provide some statistics when the event ends.

This first part is dedicated to the creation of a Node application. We will see step by step how to install Node on a Linux environment and setup a HTTP server with WebSocket support.

Introduction to Node

What is Node ?

Node is an open source toolkit for developing server side applications based on the V8 JavaScript engine. Like Node, V8 is written in C++ and is mostly known for being used in Google Chrome.

Node is part of the Server Side JavaScript environnement and extend JavaScript API to offer usual server side functionalities. Node base API can be extended by using the CommonJS module system.

Node has be created by Ryan Dahl in February 2009 and has since become very popular. It’s spirit is similar to Twisted for Python and EventMachine for Ruby.

The project is run by Joyent (the company employing Ryan Dalh) who lunched a cloud hosting service for node applications (no.de).

Node’s Goal ?

It’s goal is to offer an easy and safe way to build high performance and scalable network applications in JavaScript.

Those goals are achieved thanks it's architecture:

Single Threaded :

Node use a single thread to run instead of other server like Apache HTTP who spawn a thread per request, this approach result in avoiding CPU context switching and massive execution stacks in memory. This is also the method used by nginx and other servers developed to counter the C10K problem.

Event Loop :

Written in C++ using the Marc Lehman’s libev library, the event loop use epoll or kqueue for scalable event notification mechanism.

Non blocking I/O :

Node avoid CPU time loss usually made by waiting for an input or an output response (database, file system, web service, ...) thanks to the full-featured asynchronous I/O provided by Marc Lehmann’s libeio library.

These characteristics allow Node to handle a large amount of traffic by handling as quickly as possible a request to free the thread for the next one.

Node has a built-in support for most important protocols like TCP, DNS, and HTTP (the one that we will focus on). The design goal of a Node application is that any function performing an I/O must use a callback. That’s why there is no blocking methods provided in Node’s API.

The HTTP implementation offered by Node is very complete and natively support chunked request and response (very useful since we are going to use the twitter streaming api) and hanging request for comet applications. The Node’s footprint for each http stream is only 36 bytes (source).

JavaScript

Being an Event Driven Language, Javascript is the most suited to develop on the Node’s “Event Loop” architecture. Node’s applications really use javascript's strengths like anonymous functions and closures.

Installing Node Package Manager :

Isaac Schlueter (another employee of Joyent) has created a really great Node package manager called NPM.
We need it because our Node application will use the Socket.IO module.

$ aptitudeinstall curl
$ curl http://npmjs.org/install.sh | sh

Once installed you simply call this command line to install the Socket.IO package.

$ npm install socket.io

About Socket.IO

Socket.IO is a powerful Node module that bring the ability to simply manage long term connections with the clients. The module is used on the server side and on the client one. The client side will automatically and seamlessly use the best communication type (based on browser capabilities) to connect to a Node server.

Our architecture use a Google Chrome client, since this browser has WebSocket protocol capability, that's the connection type Socket.IO will choose.

But what happen if we decided to create a Mozilla Firefox extension where WebSocket support is disable ? Well, by using Socket.IO, long polling connection will automatically be used and there is no need to change anything in our code on both client and server sides.

Two more lines, that’s the only things we need to add to attach the Socket.IO module to the http server.

You might have noticed that the callback function of the http.createServer() has been removed, this is due to our use case where the server does not serve data to the client when they make a request but directly push data when an event is raised.

We can put a callback to the io.listen() method, it will be fired each time a request comes from the Socket.IO client side library.

Twitter tracker module

To retrieve the tweets about the “What’s Next” event we need to use the Twitter Streaming API. The API will return us in real time all the tweets mentioning “WsN_Paris” and "zenika".

In order to do that, we are going to create a simple module that will connect to the Twitter API and emit an event each time it found a new tweet related to our topic.

Connection to the API require a valid twitter account. The module will encode your credentials and add them to the request header. We do not need to use OAuth to use the Twitter streaming API.

Create a new file called twitter.js and add the following content :

// Loading required modules for the modulevar http = require('http'),// Provide the createClient() method used to create the request.
buffer = require('buffer').Buffer,// Used for credentials encoding.
EventEmitter = require('events').EventEmitter; // Used for event emission.// The tracker constructor.// username & password are your twitter credentials.// Topics are the subject(s) you want to track (separated by comma).var TwitterTracker = exports.TwitterTracker=function(username, password, topics){// Encoding credentials to base64.this.auth=new Buffer(username +':'+ password).toString('base64');
// Adding basic authentication to the header with our credentials.this.header={'Authorization':'Basic '+this.auth,'Host':'stream.twitter.com'};
this.topics= topics;
}// Implementing the observer pattern thanks to Node's EventEmitter object.// This will allow our TwitterTracker object to emit some events.
TwitterTracker.prototype=new EventEmitter;
// Track method definition.// Calling the track method will send the request to the twitter api and emit event when a new tweet arrives.
TwitterTracker.prototype.track=function track(){// Creation of the http client used to make the request.var twitter = http.createClient(80,'stream.twitter.com'),// Creation of the twitter request, we pass the http method, the api link with our topics and the header.
request = twitter.request('GET','/1/statuses/filter.json?track='+this.topics,this.header),// We get a reference to our current instance to send event in the callback.
tracker =this;
// Waiting for a response from our request to the api.// This event is fired only once.
request.on('response',function(response){
response.setEncoding('utf8');
var body ='';
// Listening to the data event.// This event will be fired every time the twitter // api send back a new chunk of json data.//// Since the twitter api is streaming us the data, we might receive // incomplete json that we will not be able to parse wihout error.//// The logic implemented in this listener is to concat chunks of data // in the body variable and detect if we have a full tweet (ending by '\r\n')// befor trying to parse.
response.on('data',function(chunk){
body = body + chunk;
var tweet, index;
if((index = body.indexOf('\r\n'))>-1){
tweet = body.slice(0, index);
body = body.slice(index +2);
if(tweet.length> 0){try{// Parsing the new tweet.
tweet = JSON.parse(tweet);
// Sending the 'tweet' event that our server.js will listen to.
tracker.emit('tweet', tweet);
}catch(error){
console.error('-!- Error while parsing tweet.');
}}}});
});
// The request to the twitter api is not made until the end() method is call.
request.end();
returnthis;
};

The only tricky part is the logic implemented in the "data" event callback. This is Twitter specific and other API are much more easy to deal with.

Using The Tracker

Now that we have a functional tweet tracker module, we include it in the server.js and start waiting for "tweet" events to broadcast them to the clients.

fantastic article. Thanks a lot for sharing. one of my faviroate things about Node.js is that Javascript is the core programming language for the browser, and writing code in node.js allows us to share code between browser and server (validation code, is a perfect example) or choose whether to execute algorithms on the browser side or the client side - and if we change our mind, we can switch back easily as well.