There’s a saying among the core Node contributors: if you’re not using streams, you’re doing Node wrong*. So I spent last week learning theÂ new Node streaming API (“streams2″)Â while building a small node app. While the documentation is pretty good, there’s not a lot of examples that are both realistic and easilyÂ digestible.

This two-part tutorial aims to build on the DailyJS’s Five Minute Guide to Streams2,Â providing real use cases for each of the different types of new streams (part 1) and the application of more complex design patterns to streams (part 2).Â If you haven’t read DailyJS’s tutorial yet, go ahead and read it now. Really, go.

You’re back? Good. You’ll notice that I use a different style of inheritance. For now, consider it personal preference and feel free to rewrite using your preferred style.

Readable Stream

Let’s start by creatingÂ a Readable stream that emits Javascript objects. This was the first custom stream I wrote, since I was buffering data in memory and then wanted to stream it through a CSV formatter and out through FTP. We’ll create a streams2Â Readable streamÂ that accepts a Javascript array and emits one element of that array at a time.

Notice that we passÂ objectMode=trueÂ as an option to the stream, which tells it we’re handling Javascript objects rather than buffers or strings.Â TheÂ sizeÂ argument toÂ _readÂ is advisory, so we’ve safely ignored it. WeÂ push each element out, one at a time. Finally, you signal the end of a Readable stream by pushingÂ null.

A jasmine-node test for the ArrayStream class will demonstrate ArrayStream usage.

As you can see, ArrayStream emits ‘data’ and ‘end’ events. In fact, ArrayStream can do everything a Readable stream can do, including being piped.

Transform Stream

Okay, so now we have a stream of objects. What if we want to CSVify the objects? We need aTransform streamÂ to convert each object into an arrayÂ representing a single ordered CSV record. This is what a streaming CSV stringifier needs as input.

Note the use of objectMode=trueÂ again. This transform stream takes in a datapoint object, loops over the fields to be present in the CSV file (i.e., keys in the datapoint object), and pushes the values of the fields into a record array. We thenÂ push the transformed record out andÂ signal completion by callingÂ callback(). If there was an error, we could pass it to the callback as callback(err).

PassThrough Stream

Now we’ll see one of the most frequent uses of the newÂ PassThrough stream: testing! Let’s write a jasmine-node test for CsvPrepStream; this test will demonstrate usage of PassThrough streams in addition to the functionality of CsvPrepStream.

As you can see, PassThrough streams make great sources of data in testing! Just write some data into it in chunks, then end the stream. It’s that easy. If you’re writing anything other than strings or buffers, just remember that it must haveÂ objectMode=true.

Note that in a real test you’d want to test all possible scenarios, such as fields in the array that aren’t present as keys in the object and keys in the object that aren’t present in the array. This merely serves as an example.

Writable Stream

The final stream we’re going to cover today is the Writable stream. Once the CsvFormatter was written, we needed a way to capture the entire CSV contents as a string for testing it. Thus,Â the StringBufferStream was born.

Conclusion

In this tutorial, we have seen real use cases for almost** all of the “streams2″ stream: Readable, Writable, Transform, and PassThrough. We’ve also seen examples of how to pass Javascript objects through your stream pipeline and how to test your custom streams with jasmine-node. In Part 2 of this write-up, we’ll cover some more advanced uses cases and apply classic design patterns to streams to solve some tricky problems. Stay tuned!

* At least that’s what I heard from a colleague who attended the last NodeConf.
**Â Once I find a good use case for the Duplex stream, I’ll write something up for it too.

Subscribe

About Cody A. Ray

I’m an inquisitive, tech-savvy, entrepreneurially-spirited dude. Currently, I’m a DevOps engineer at PEAK6, an entrepreneurial investment firm in downtown Chicago. This is my personal blog. Giving true meaning to the origin of the term, my blog is a catalog of my thoughts on various matters, ranging from technology tutorials to social commentary. My goal is to create insightful, […]more →