In my last blog entry, I shared some nodejs-code to read CouchDB's _changes feed and publish the data to a website. In order to update the page in a continous fashion, I used socket.io which provides a nifty abstraction across server- to client-side transports — for example, websockets and ajax longpoll.

Full-throttle

When we tested the code for a few days over the weekend, the largest issue we ran into was that the stream moved too fast. In fact it moved so fast, we couldn't read anything and were at risk of getting a seizure when we watched the page for too long.

Certainly awesome from one point of view — people are using the website — but it also led to the next objective: I had to find a way to throttle broadcasting to the client. Here's how!

Decisions, decisions!

So in the first iteration of the project we read _changes with ?include_docs=true.

This combination makes it incredibly easy to get all data from BigCouch/CouchDB. However, when you're not looking to broadcast all data, but only a fraction of it at a set interval, it becomes pointless to add the extra burdon on the cluster which ?include_docs=true brings along.

Moving forward, we decided to not use ?include_docs=true anymore (Hat tip to Adam of Cloudant for consultation!) and instead use the IDs provided by _changes and request individual documents on demand.

The interval to broadcast data to the realtime stream for the user is set at every 500 milliseconds, which equals to more or less two requests per second.