Node.js Stream Playground

This summer I had the opportunity to attend NodeConf and it was a fantastic experience. I really appreciated how every session was a hands-on coding session: I felt like I walked away knowing how to put a bunch of advice directly into practice.

One of my favorite sessions was the one run by James Halliday and Max Ogden exploring Node streams. Specifically they sat us down and had us run through the amazing Stream Adventure program. It’s an interactive series of exercises that are designed to help you understand streams better. Seeing interesting ways of using streams in practice was quite eye opening for me. (I also recommend checking out the awesome Stream Handbook if streams are totally foreign to you)

I had made limited use of streams before (namely with the request and fs modules) but I was still early on. Over the past few months I’ve spent more time researching Node modules that make good use of streams and have tried to apply them in my various side projects. In theory I absolutely love streams: they are the chainable piping amazingness that exists with UNIX and I aspired to achieve with jQuery. In practice many Node stream modules are poorly written, poorly documented, out of date, or just obscure. So while theoretically using Node streams should be really easy in practice it becomes quite challenging.

In my explorations I was starting to build up a pile of streaming Node modules that I *knew* worked correctly (or, at least, as I expected them to). Namely: They supported streams in Node 0.10+ and they supported being .pipe()‘d to and from (where applicable). I realized that in a perfect world Node streams truly are like Lego bricks – it becomes quite easy to snap a few of them together to build a chained flow for your data to pass through.

With that in mind I built an exploration that I hope will help others to understand the simplicity and power of Node streams: The Node.js Stream Playground.

The Node.js Stream Playground was created to help Node.js developers better understand how streams work by showing a number of use cases that are easily plug-and-play-able. Additionally detailed logging is provided at every step to help users better understand what events the streams are emitting and exactly what their contents are.

How to Use the Playground

I hope you’ll get the most out of the playground by exploring concepts for yourself – looking at what happens when you pipe different streams to each other, looking at the data being logged, and attempting to understand (for yourself) what exactly is happening and why. Note that all of the code being generated is valid Node code and can be copy-and-pasted and run on your own computer (assuming you have the appropriate NPM modules installed).

I should note that this isn’t the end-all-and-be-al of Node stream education: There is still a lot to learn with regard to error handling, back pressure, and all the intricate stream concepts that exist. I hope to write some more on these concepts some day.

With that being said here are some good examples of some of the actions that you can do:

Note that the initial data that is coming out of the Read File is just an array of numbers – this is to be expected. In reality it’s a Node Buffer holding all of the raw binary data. It’s not until after we Un-Gzip the file that we start to have data that’s in a more usable form.

Converting a TSV to HTML

In this case we’re going to manually parse the TSV (without headers) just to show you some of the string operations provided by event-stream.

Select Read File and input/people.tsv.

Select Split Strings.

Select Split Strings into Array.

Select Convert Array w/ Sprintf (with the default table row HTML).

Select Join Strings (this will insert an endline between all the individual table rows).

Select Concat Strings (this will combine all the individual table rows and endlines into a single string).

You should have the same output as the last run. A lot is happening here but even will all of these steps Node streams still makes it relatively easy to complete it all. It’s at this point that you can truly start to see the power and expressiveness of streams.

Adding in New Streams

If you’re interested in extending the playground and adding in new pluggable stream “blocks” you can simply edit blocks.js and add in the stream functions. A common stream block would look something like this:

The property name is the full title/description of the stream. The arguments to the function are variables that you wish the user to populate. The comments immediately following the argument names are the default values (you can provide multiple values by separating them with a |).

The streams are split into 3 types: “Readable”, “Writable”, “Transform” (in that they read content, modify it, and pass it through). Typically it is assumed that “Readable” streams will be the first ones that you can choose in the playground, “Writeable” streams will end the playground, and everything else is just “.pipe()able”.

Running Your Own Server

After downloading the code (either from Github or NPM). Be sure to run npm install to install all the dependencies. You can then use just node app.js to run a server – or if you wish to run something more robust you can install naught and then run npm start.

WARNING I have no idea how robust the application’s security is. This application is generating and executing code on the user’s behalf (although it is not allowing arbitrary code to be executed). Feel free to run it on a local server or, if you feel confident in the code, run it on your own server. At the moment I’m running it on a standalone server with nothing else on it, just in case.

Feedback Welcome!

I’d love to hear about how people are using streams and if this tool has been helpful for your understanding how streams work. Let me know if there are particular stream modules that you really like and – if possible – try and add them to the stream playground so that others can experience them as well!

I have to agree that one of the best sessions I’ve ever had the chance to be in was the Streams Adventures session. It was a really great way to get people up and running and working on their own, and being available if you got stuck.

Thanks for sharing this awesome tool, and it’s always cool to see people walk away from a learning experience and create another learning experience.