Streams in node are awesome. I’ve heard one of the core node contributors say, “if you’re not using streams, you’re doing it wrong”. Why are streams awesome? I’m glad you asked.

improve performance for large data sets—stream from source to response without buffering it all in memory

improve perceived responsiveness—start responding as soon as any data is available, rather than waiting for it all

clean service composition pattern—make a request to another service and stream its response back to the user, filtering or modifying it in some way in the process

However, streaming in node isn’t straightforward. This is more true with the introduction of Stream2, aka “new streams”, since module writers try to support both old and new streams.

I ran into this challenge the other day. After hours of debugging, I realized that mixing stream modes was the cause and I somehow stumbled onto a stupid simple solution.

The goal was to stream Javascript arrays into node-csv and then use node-ftp to upload the resulting CSV to a file. For some reason, though, the first line or two were always missing from the created file.

Notice its missing the first row? If you tell the CSV module to add a header, it’ll be missing the header and the first data row. This happens regardless of whether or not you write into the data source before the FTP connection. (You can see that because “d,e,f” is in the output file.)

It turns out that the CSV module is emitting the data before the FTP module is ready for it. After several hours of banging my head on the table, I surmised that the CSV module and the FTP module try to handle both old and new streams, and do so in incompatible ways. A couple more hours of banging, and I stumbled onto this solution.

Really, that’s it. My theory is that this works because the built-in PassThrough stream is compatible with both the old and new streaming modes, and by playing middleman it can help “translate” between the CSV and FTP modules.

Subscribe

About Cody A. Ray

I’m an inquisitive, tech-savvy, entrepreneurially-spirited dude. Currently, I’m a DevOps engineer at PEAK6, an entrepreneurial investment firm in downtown Chicago. This is my personal blog. Giving true meaning to the origin of the term, my blog is a catalog of my thoughts on various matters, ranging from technology tutorials to social commentary. My goal is to create insightful, […]more →