Main

Real Time Web Analytics with Node.js and Socket.IO

I’ve always enjoyed building real-time applications. A few years ago, I built the Java back-end for the Tour of California “Tour Tracker”, and over the years, I have also been involved in several real-time Trader Desktop projects that I documented in this blog. So I thought it would be nice to explore the process of building a real-time application with Node.js and Socket.IO.

As an example, we’ll build a simple web analytics dashboard that shows pages being accessed on a web site in real-time. Kind of like the real-time view in Google Analytics.

The basic flow of the app is as follows:

The instrumented web application sends a message to the server every time there is a page change within the app.

Navigate through the application: the dashboard should display each page view.

In the remaining of this article we will go through the steps of building the application from scratch.

First Iteration: Basic Implementation

Node.js and Socket.IO make it easy to implement this type of real-time messaging system. Here is a simple (yet fully functional) implementation of the application:

The Server

In this first implementation, Node.js doesn’t serve the application’s HTML pages. Pages loaded from any domain (or even from the file system) can connect to our message server to send or listen to “pageview” messages.

The Instrumented Web App

Here is an example of an instrumented page with a script that sends a message to the server every time the page is loaded.

As mentioned above, this page can be loaded from any domain or from the file system.

The Real Time Dashboard

Here is a bare-bones implementation of the real-time dashboard application that receives “pageview” messages from the server.

That’s basically all you need to start building a sophisticated user interface with real-time charts, counters, maps, etc.

Testing the Application

Make sure you have Node.js and Socket.IO installed

Start the Node.js server

node server

Double-click dashboard.html in Finder (Mac) or Explorer (Windows) to open the page in your default browser.

Double-click simplepage.html in Finder or Explorer to open the page in your default browser (in a separate window).

Refresh simplepage.html repeatedly: the dashboard should display each page view.

Second Iteration: Adding Authorization

The solution described above works fine, but there are some potential security issues we need to address:

Any application (from any domain) can send a message to our server.

Any application (from any domain) can listen to the messages pushed by our server.

Let’s refine our solution to make sure that only authorized clients can connect to our server. Socket.IO provides two different authorization methods: global authorization and namespace authorization (more info). In both cases, the handshakeData object provides information (headers, IP address, xdomain, etc) that is useful to implement your authorization logic and determine whether or not you want to allow the client to connect. In this application, we will use global authorization and reject all the cross-domain connection attempts:

There are many other ways you can secure your connections, but that discussion is beyond the scope of this article.

Rejecting cross-domain connections means that the application pages now also have to be served by the Node.js server. To put our new authorization policy into practice, let’s add the real-time web analytics capability to the Node Cellar application that I shared in my previous post.

The Server

The server now serves the HTML pages and hosts Socket.IO.

The Instrumented Web App

The following script has been added to index.html to send a “pageview” message to the server every time the page changes.

Notes:

We actually don’t need to use Socket.IO for this part of the application. The client sends a “pageview” message to the server and doesn’t listen for messages from the server. In other words, this is a traditional client-to-server one-way communication that could be implemented with a traditional Ajax request.

The onhashchange event is used because Node Cellar is a “Single Page Application”. The entire application runs within a single page (index.html): views are created dynamically at the client-side and injected into or removed from the DOM as you navigate through the application. Only the hash part of the URL changes (without a page refresh) to reflect the current state of the application.

The Real Time Dashboard

The final version of the dashboard is shown in the screenshot at the top of this post. The Socket.IO code didn’t change much since our initial implementation. The final code for dashboard.html is available here.

Testing the Application

Make sure you have Node.js and Socket.IO installed

Download the source code

Uncomment the Socket.IO related script in the head of public/index.html

The code is explicit, it save the url of the external links the user clicks tracking its exit.

rinku das

when we are refreshing the dashboard, there is no pageview record listed..even my client pages are opened..how to solve this problem, so that after refreshing the dashboard we can get previous result as well as updated result?plz ignore my english and programming knowledge..

I’m surprised how little code it takes to get something together with this framework. This is coming from someone who has suffered through developing with RoR, as well as a few of the countless unnamed frameworks for PHP.

I now have an excellent starting point for building a REAL real-time app with Node.JS. I’m surprised how easy it is….

I’m designing a site with PHP Zend Framework, plugged node.js into it to support instant notifications on the messaging module,
to share the session centrally kept the PHP Session in redis and use it for authentication at node.js end.
Included the node.js client javascript on all web pages to get notification bubble across the site.
Issue : If I navigate from a page to another, the socket connection disconnects and connects again.
any suggestions to overcome this.

daniel

Very nice addon!!

I try to host a nodejs application on heroku,

how can i modify the socket.io path (http://localthost),
that i works an the heroku instance?

Kind regards
daniel

Beraki

Nice Tutorial but I wanted to know which free nodejs server I can get to taste my product before testing.

Introducing web application development with Node.js, this book walks you through developing of a full-fledged chat system built with socket.io and introducing all the concepts of socket.io and its usage in the process.

Jibran

Great blog post. I have a question about some general dedicated server implementation. If I want to use a server instead of “localhost” how would I go about doing that. I have looked online for a general solution and most people suggest placing the URL into the area that reads “io.connect “. In short I replaced the socket variable to var socket = io.connect(‘http://mydoamin.com’);”. However this does not work, the browser throws an error saying that it cant find socket.io library. Any Ideas?

of course like your website however you have to take a look at the spelling on several of your posts.
Many of them are rife with spelling problems and I to find it very bothersome to tell the truth
nevertheless I will definitely come back again.

Really nice App. Thank you. I am a beginner and it will help to start learning backbone and Bootstrap. I have some ideas I will try to learn and do my webapp on my own. I am wondering why you didn’t commercialize this. Small companies can buy this right and is very useful for them just like any time tracking app.

Need small Info: Based on backbone+Bootstrap, are there any open srcd/example templates where a simple app that has support for user login/signups. I want to create an app that allows user to login and enter some details for app to work. One cant access others data. Any pointers how to go about ?? Any one else can also help me.

I have been browsing online more tthan 2 hours today, yet I never
found any interestging article like yours. It’s pretty
worth enough for me. In my view, if all webmasters and
bloggers made good content as you did, the net
will be much more userul than ever before.

Many claim that it is too soon for another updated and upgraded version
of the i – Pad. Protecting your investment is important regardless of what you buy;
it is a good idea to get a good support and ticket system even with
the Apple Mac – Book Pro. Afloat also offers many other additional little niceties,
like making windows transparent with a mouse gesture or multi-touch trackpad.

Great tutorial….
I have failed to understand this error…. while trying to run the serverwithanalytics.js file. It is related to socket.io module… especially the undefined is not function
io.configure(function () {
^
TypeError: undefined is not a function
at Object. (G:\GOOGLE DRIVE\CODE POLISHER\Nodej\tuts\nodecellar-master\serverwithanalytics.js:20:4)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)
at node.js:814:3