Gulp Under the Hood: Building a Stream-based Task Automation Tool

Front-end developers nowadays are using multiple tools to automate routine operations. Three of the most popular solutions are Grunt, Gulp and Webpack. Each of these tools are built on different philosophies, but they share the same common goal: to streamline the front-end build process. For example, Grunt is configuration-driven while Gulp enforces almost nothing. In fact, Gulp relies on the developer writing code to implement the flow of the build processes - the various build tasks.

When it comes to choosing one of these tools, my personal favorite is Gulp. All in all it’s a simple, fast and reliable solution. In this article we will see how Gulp works under the hood by taking a stab at implementing our very own Gulp-like tool.

Gulp API

These four simple functions, in various combinations offer all the power and flexibility of Gulp. In version 4.0, Gulp introduced two new functions: gulp.series and gulp.parallel. These APIs allow tasks to be run in series or in parallel.

Out of these four functions, the first three are absolutely essential for any Gulp file. allowing tasks to be defined and invoked from the command line interface. The fourth one is what makes Gulp truly automatic by allowing tasks to be run when files change.

Gulpfile

It describes a simple test task. When invoked, the file test.txt in the current working directory should be copied to the directory ./out. Give it a try by running Gulp:

touch test.txt # Create test.txt
gulp test

Notice that method .pipe is not a part of Gulp, it’s node-stream API, it connects a readable stream (generated by gulp.src('test.txt')) with a writable stream (generated by gulp.dest('out')). All communication between Gulp and plugins are based on streams. This lets us write gulpfile code in such an elegant way.

Meet Plug

Now that we have some idea of how Gulp works, let’s build our own Gulp-like tool: Plug.

We will start with plug.task API. It should let us register tasks, and tasks should be executed if the task name is passed in command parameters.

This will allow tasks to be registered. Now we need to make this task executable. To keep things simple, we will not make a separate task launcher. Instead we will include it in our plug implementation.

All we need to do is run the tasks named in command line parameters. We also need to make sure we attempt to do it in the next execution loop, after all tasks are registered. The easiest way to do it is run tasks in a timeout callback, or preferably process.nextTick:

Note that Gulp runs subtasks in parallel. But to keep things simple, in our implementation we are running subtasks sequentially. Gulp 4.0 allows this to be controlled using its two new API functions, which we will implement later in this article.

Source and Destination

Plug will be of little use if we don’t allow files to be read and written to. So next we will implement plug.src. This method in Gulp expects an argument that is either a file mask, a filename or an array of file masks. It returns a readable Node stream.

Note that we use objectMode: true, an optional parameter here. This is because node streams work with binary streams by default. If we need to pass/receive JavaScript objects via streams, we have to use this parameter.

As you can see, we created an artificial object:

{
name: path, //file name
buffer: data //file content
}

… and passed it into the stream.

On the other end, plug.dest method should receive a target folder name and return a writable stream which will receive objects from .src stream. As soon as a file object will be received, it will be stored into the target folder.

Gulp itself works about the same way, but instead of our artificial file objects it uses vinyl objects. It is much more convenient, as it contains not just the filename and content but additional meta information as well, such as the current folder name, full path to file, and so on. It may not contain the entire content buffer, but it has a readable stream of the content instead.

Vinyl: Better Than Files

There is an excellent library vinyl-fs that lets us manipulate files represented as vinyl objects. It essentially lets us create readable, writable streams based on file mask.

We can rewrite plug functions using vinyl-fs library. But first we need to install vinyl-fs:

npm i vinyl-fs

With this installed, our new Plug implementation will look something like this:

Now on each change of test.txt, the file will be copied into the out folder with its name changed.

Series vs Parallel

Now that all the fundamental functions from Gulp’s API is implemented, let’s take things one step further. The upcoming version of Gulp will contain more API functions. This new API will make Gulp more powerful:

gulp.parallel

gulp.series

These methods allow the user to control the sequence in which tasks are run. To register subtasks in parallel gulp.parallel may be used, which is the current Gulp behavior. On the other hand, gulp.series may be used to run subtasks in a sequential manner, one after another.

Assume we have test1.txt and test2.txt in the current folder. In order to copy those files to out folder in parallel let us make a plugfile:

We rely on node stream ‘end’ which is emitted when a stream has processed all messages and is closed, which is an indication that the subtask is complete. With async.js, we do not have to deal with a big mess of callbacks.

To try it out, let us first run the subtasks in parallel:

node plugFile.js test-parallel

task test-parallel is started
task subTask1 is started
task subTask2 is started
stream subTask2 is ended
stream subTask1 is ended
done

And run the same subtasks in series:

node plugFile.js test-series

task test-series is started
task subTask1 is started
stream subTask1 is ended
task subTask2 is started
stream subTask2 is ended
done

Conclusion

That’s it, we have implemented Gulp’s API and can use Gulp plugins now. Of course, do not use Plug in real projects, as Gulp is more than just what we have implemented here. I hope this little exercise will help you understand how Gulp works under the hood and let us more fluently use it and extend it with plugins.

About the author

Mikhail is a software engineer looking for challenging projects. He has completed several web-based projects with Node.js/Go (back-end) and JavaScript SPA (front-end). He has experience working with React.js, RIOT.js AngularJS UI frameworks, Flux/Redux architecture, and back-end development (architecture, testing, deployment, monitoring, reporting, etc.). He's mostly looking for front-end development gigs, but can help with back-end as well. [click to continue...]

Mikhail is a software engineer looking for challenging projects. He has completed several web-based projects with Node.js/Go (back-end) and JavaScript SPA (front-end). He has experience working with React.js, RIOT.js AngularJS UI frameworks, Flux/Redux architecture, and back-end development (architecture, testing, deployment, monitoring, reporting, etc.). He's mostly looking for front-end development gigs, but can help with back-end as well.