Four ways of handling asynchronous operations in node.js

Posted six years ago

Javascript was not designed to do asynchronous operations easily. If it were,
then writing asynchronous code would be as easy as writing blocking code.
Instead, developers in node.js need to manage many levels of callbacks.

Today, we will examine four different methods of performing the same task
asynchronously, in node.js. We will read in all the files in the current
folder,

In parallel, using callback functions

Sequentially, using callback functions

In parallel, using promises

Sequentially, using promises

This will help you decide which to use for your particular situation. It is simply a matter of taste. If you want to know which is the best method -- the absolute best way to go -- is probably to switch to Golang.

Reading the files in parallel using callbacks

Our toy problem is to read all of the files in the current folder.

With our toy problem, we can't do everything in parallel. To read the files in
the folder, we first need to know which files are in the folder. Thus we start
with the readdir() function. We wait for the operation to complete. Then, for
each file, we use the readFile() to get the contents of the file.

Here are the documentation for the functions, from node.js.

fs.readdir(path, [callback])

Asynchronous readdir(3). Reads the contents of a directory. The callback gets two arguments (err, files) where files is an array of the names of the files in the directory excluding '.' and '..'.

fs.readFile(filename, [encoding], [callback])

Asynchronously reads the entire contents of a file.

The callback is passed two arguments (err, data), where data is the contents of the file.

If no encoding is specified, then the raw buffer is returned.

All of the readFile() operations happen at once, and then we wait for the
results to come in. We simply count how many times a readFile() operation has completed. When all of the files have been read, we
know we are done.

Reading the files sequentially using callbacks

It is usually most efficient to to the above. Order the computer to do everything at once, and let the operating system sort it out. But that's not always what you want. Sometimes you need to impose an order and do things sequentially.

Here is an example of reading each file one at a time. The for loop is gone. It is replaced by a recursive function. The function checks to see if it has reached the last file. If so, it is done. Otherwise, it calls itself to process the next file in the list.

Reading the files in parallel using Promises

A promise (also known a future, and sometimes a channel) is a concept from the 1970's that has recently become popular for Javascript programming. Promises are implemented in the node.js module promise. This module is not included unless you add it.

When you call a function and expect a return value, and the value is not yet available, the function instead returns a promise. The caller can then store the promise for later or schedule a subsequent operation when it completes. Promises can be seen as a more specialized form of node.js's EventEmitter, where the only two events are reject or resolve. Instead of using "on" to listen for the events, we use "then".

Promises may be easier to deal with for some people, because functions that return promises are harder to misuse. You could forget whether a callback belongs in the 3rd or 4th parameter, but you can't make that mistake with a return value. Another advantage is that they encapulate the recursive function loop above. You can easily construct a super-promise from an array of promises, what will resolve only when each of its members resolve. That's we do in the code below, with Promise.all()

Reading the files sequentially using Promises

By default, promises don't support sequential operations very well. But we can build on them using a PromiseSequence, which adds the ability to define a series of steps and loops, which are performed sequentially.

Here is the program again, reading a file. Instead of indenting the code by many levels, we are able to write it in more of a sequential style. Also, the above examples had two places where errors had to be handled. With a promise sequence, the errors for any operation in the sequence are handled in one place.

We first add the readdir() operation to the sequence, and then add a loop() to the sequence. The loop executes repeatedly until exitLoop() is called. Since there are no further step, the argument to exitLoop resolves the promise and the program ends.

Why does my brain hurt?

I have been programming for over two decades. I know lots of languages, but only Javascript makes my brain hurt when I have to do simple things. If you have a lot of asynchronous operations to perform, and you have the choice, please consider your language selection very carefully.

Sometimes you need complete control over the server, and don't want to pay $20 to $40 a month for a VPS. In this article, I'll describe step by step how to set up a home web server using Ubuntu, capable of handling modest spikes in traffic.

I just spent several hours debugging clipboard copy of a DIB image. I could copy from my application, and paste into Paint. I could paste into Word. But if I pasted into WordPad, nothing showed up. If I pasted into GIMP, it crashed.

Back in 2007, I created a rhyming engine based on the public domain Moby pronouncing dictionary. It simply reads the dictionary and looks for rhyming words by comparing the suffix of the words' pronunciations. Since that time, I have made some improvements.

Most people, having already paid $2000.00 of their hard earned money, and then having flown, driven, or otherwise travelled to Boston to attend a conference, and then having paid an additional $250/night plus $33/night parking and "tourism taxes" to the Seaport Hotel -- most people, after all this, are unlikely to say that it was a waste of time and they should have stayed home watching the remaining salvaged episodes of Doctor Who on Netflix.