README.md

Builder

Builder takes your npm tasks and makes them composable, controllable from
a single point, and flexible.

npm is fantastic for controlling tasks (via scripts) and general project
workflows. But a project-specific package.json simply doesn't scale when
you're managing many (say 5-50) very similar repositories.

Enter Builder. Builder is "almost" npm, but provides for off-the-shelf
"archetypes" to provide central sets of package.jsonscripts tasks, and
dependencies and devDependencies for those tasks. The rest of this page will
dive into the details and machinations of the tool, but first here are a few of
the rough goals and motivations behind the project.

Single Point of Control: A way to define a specific set of tasks /
configs / etc. for one "type" of project. For example, we have an
ever-expanding set of related repos for our
Victory
project which all share a nearly-identical dev / prod / build workflow.

Flexibility: There are a number of meta tools for controlling JavaScript
workflows / development lifecycles. However, most are of the "buy the farm"
nature. This works great when everything is within the workflow but falls
apart once you want to be "just slightly" different. Builder solves this by
allowing fine grain task overriding by name, where the larger composed tasks
still stay the same and allow a specific repo's deviation from "completely off
the shelf" to be painless.

You Can Give Up: One of the main goals of builder is to remain very
close to a basic npm workflow. So much so, that we include a section in this
guide on how to abandon the use of Builder in a project and revert everything
from archetypes back to vanilla npmpackage.jsonscripts, dependencies
and devDependencies.

A Few "Nice to Haves" Over npm run <task>: Setting aside archetypes and
multi-project management, builder provides cross-OS compatible helpers for
common task running scenarios like concurrent execution (concurrent) and
spawning the same tasks in parallel with different environment variables
(env). It also provides useful controls for task retries, buffered output,
setup tasks, etc.

Overview

At a high level builder is a tool for consuming package.jsonscripts
commands, providing sensible / flexible defaults, and supporting various scenarios
("archetypes") for your common use cases across multiple projects.

Builder is not opinionated, although archetypes are and typically dictate
file structure, standard configurations, and dev workflows. Builder supports
this in an agnostic way, providing essentially the following:

NODE_PATH, PATH enhancements and module patterns to run, build, import
from archetypes so task dependencies and configurations don't have to be
installed directly in a root project.

A task runner capable of single tasks (run) or multiple concurrent tasks
(concurrent).

An intelligent merging of package.jsonscripts tasks.

... and that's about it!

Usage

To start using builder, install and save builder and any archetypes you
intend to use. We'll use the builder-react-component archetype as an
example.

Note: Most archetypes have an ARCHETYPE package and parallel
ARCHETYPE-dev npm package. The ARCHETYPE package contains almost
everything needed for the archetype (prod dependencies, scripts, etc.) except
for the devDependencies which the latter ARCHETYPE-dev package is solely
responsible for bringing in.

Global Install

For ease of use, one option is to globally install builder and locally install
archetypes:

To make these changes permanent, please see this multi-OS article on
changing the PATH variable: https://www.java.com/en/download/help/path.xml
(the article is targeted for a Java executable, but it's analogous to our
situation). You'll want to paste in ;node_modules\.bin at the end ornode_modules\.bin; at the beginning of the PATH field in the gui. If there
is no existing PATH then add a user entry with node_modules\.bin as a value.
(It is unlikely to be empty because an npm installation on Windows sets the
user PATH analogously.)

Full Path Invocation

Or you can run the complete path to the builder script with:

Mac / Linux

node_modules/.bin/builder <action><task>

Windows

node_modules\.bin\builder <action><task>

Configuration

After builder is available, you can edit .builderrc like:

---
archetypes:
- builder-react-component

to bind archetypes.

... and from here you are set for builder-controlled meta goodness!

Builder Actions

Display general or command-specific help (which shows available specific flags).

builder concurrent

Run multiple tasks from script concurrently. Roughly analogous to
npm run <task1> & npm run <task2> & npm run <task3>, but kills all processes on
first non-zero exit (which makes it suitable for test tasks), unless --no-bail
is provided.

$ builder concurrent <task1><task2><task3>

Flags:

--tries: Number of times to attempt a task (default: 1)

--setup: Single task to run for the entirety of <action>.

Note: The --setup task is run at the start of the first main task
to actually run. This may not be the first specified task however,
as pre tasks could end up with main tasks starting out of order.

--queue: Number of concurrent processes to run (default: unlimited - 0|null)

--[no-]buffer: Buffer output until process end (default: false)

--[no-]bail: End all processes after the first failure (default: true)

Note that tries will retry individual tasks that are part of the concurrent
group, not the group itself. So, if builder concurrent --tries=3 foo bar baz
is run and bar fails twice, then only bar would be retried. foo and baz
would only execute once if successful.

builder envs

Run a single task from script concurrently for each item in an array of different
environment variables. Roughly analogous to:

Note: The --setup task is run at the start of the first main task
to actually run. This may not be the first specified task however,
as pre tasks could end up with main tasks per environment object
starting out of order.

--queue: Number of concurrent processes to run (default: unlimited - 0|null)

--[no-]buffer: Buffer output until process end (default: false)

--[no-]bail: End all processes after the first failure (default: true)

Note: The environments JSON array will overwrite existing values in the
environment. This includes environment variables provided to / from builder
from things such as npmconfig and the --env/--env-path flags.

So, for example, if you invoke builder with:

$ builder envs <task>'[{"FOO": "ENVS"}]'--env='{"FOO": "FLAG"}'

The environment variable FOO will have a value of "ENVS" with the single
environment object array item given to builder envs overriding the --env
flag value.

Setup Task

A task specified in --setup <task> will have the following flags apply to
the setup task as apply to the main task:

--env

--env-path

--quiet

--log-level

The following flags do not apply to a setup task:

-- custom flags

--tries

--expand-archetype

--queue

--buffer

That said, if you need things like --tries, etc., these can be always coded
into a wrapped task like:

running builder concurrent foo bar would run all of the above tasks at
the appropriate lifecycle moment.

Note that things like a --queue=NUM limit on a concurrent task will have
all of the pre, main, and post task need to finish serial execution before
the next spot is freed up.

The --bail flag applies to all of a single tasks pre, main, and post
group. So if any of those fail, it's as if the main task failed.

Builder Flags During Pre and Post

Applicable Flags

When executing a <task> that has pre<task> and/or post<task> entries, the
following execution flags do apply to the pre|post tasks.

--env

--env-path

--quiet

--log-level

--expand-archetype

These flags have mixed application:

--queue: Applies for concurrent, but not envs. The flag is invalid for
run.

--buffer: Applies for concurrent, but not envs. The flag is invalid for
run.

--bail: Applies for concurrent, but not envs. The flag is invalid for
run. A pre<task>, <task>, and a post<task> are treated as a group, so
a failure of any short-circuits the rests and ends with failures. But with
--bail=false a failure doesn't stop execution of the other groups.

The following flags do not apply to pre/post tasks:

-- custom flags

--tries

--setup: A task specified in --setup <task> will not have pre|post
tasks apply.

We will explain a few of these situations in a bit more depth:

Custom Flags

The special -- flag with any subsequent custom flags to the underlying task
are only passed to the the main <task> and not pre<task> or post<task>.
The rationale here is that custom command line flags most likely just apply to
a single shell command (the main one).

By contrast, the various other Builder-specific flags that can be applied to a
task like --env, etc., will apply to pre|post tasks, under the
assumption that control flags + environment variables will most likely want to
be used for the execution of all commands in the workflow.

Task Prefix Complexities

For the above example, if you have a task named preprefoo, then running
fooor even prefoo directly will not run preprefoo. Builder
follows npm's current implementation which is roughly "add pre|post tasks
to current execution as long as the task itself is not prefixed with
pre|post". (Note that yarn does not follow this logic in task execution).

Custom Flags

Just like npm run <task> [-- <args>...],
flags after a -- token in a builder task or from the command line are passed
on to the underlying tasks. This is slightly more complicated for builder in
that composed tasks pass on the flags all the way down. So, for tasks like:

"scripts": {
"down":"echo down",
"way":"builder run down -- --way",
"the":"builder run way -- --the",
"all":"builder run the -- --all"
}

We can run some basics (alone and with a user-added flag):

$ builder run down
down
$ builder run down -- --my-custom-flag
down --my-custom-flag

If we run the composed commands, the -- flags are accumulated:

$ builder run all
down --way --the --all
$ builder run all -- --my-custom-flag
down --way --the --all --my-custom-flag

Expanding the Archetype Path

In npm v2 this wasn't a problem because dependencies were usually nested. In
npm v3, this all changes with aggressive
flattening of dependencies. With flattened
dependencies, the chance that the archetype and its dependencies no longer have
a predictable contained structure increases.

Thus, commands like the above succeed if the installation ends up like:

Then builder can still find webpack due to its PATH and NODE_PATH
mutations. But an issue arises with something like a postinstall step after
this flattening in that the current working directory of the process will be
PATH/TO/node_modules/<a module>/, which in this flattened scenario would
not find the file:

node_modules/<archetype>/config/webpack/webpack.config.js

because relative to node_modules/<a module>/ it is now at:

../<archetype>/config/webpack/webpack.config.js

To address this problem builder has an --expand-archetype flag that will
replace an occurrence of the specific node_modules/<archetype> in one of the
archetype commands with the full path to the archetype, to guarantee
referenced files are correctly available.

The basic heuristic of things to replace is:

^node_modules/<archetype>: Token is very first string.

[\s\t]node_modules/<archetype>: Whitespace before token.

['"]node_modules/<archetype>: Quotes before token.

Note that the path coming back from the underlying
require.resolve(module) will likely be escaped, so things like
whitespace in a path + quotes around it may not expand correctly.

Some notes:

The only real scenario you'll need this is for a module that needs to run
a postinstall or something as part of an install in a larger project.
Root git clone projects controlled by an archetype should work just fine
because the archetype will be predictably located at:
node_modules/<archetype>

The --expand-archetype flag gets propagated down to all composed builder
commands internally.

The --expand-archetype only expands the specific archetype string for its
own commands and not those in the root projects or other archetypes.

The replacement assumes you are using / forward slash characters which
are the recommended cross-platform way to construct file paths (even on
windows).

The replacement only replaces at the start of a command string or after
whitespace. This means it won't replace ../node_modules/<archetype> or
even ./node_modules/<archetype>. (In the last case, just omit the ./
in front of a path -- it's a great habit to pick up as ./ breaks on Windows
and omitting ./ works on all platforms!)

Tasks

The underlying concept here is that builderscript commands simply are
npm-friendly package.jsonscript commands. Pretty much anything that you
can execute with npm run <task> can be executed with builder run <task>.

Builder can run 1+ tasks based out of package.jsonscripts. For a basic
scenario like:

{
"scripts": {
"foo":"echo FOO",
"bar":"echo BAR"
}
}

Builder can run these tasks individually:

$ builder run foo
$ builder run bar

Sequentially via || or && shell helpers:

$ builder run foo && builder run bar

Concurrently via the Builder built-in concurrent command:

$ builder concurrent foo bar

With concurrent, all tasks continue running until they all complete or
any task exits with a non-zero exit code, in which case all still alive tasks
are killed and the Builder process exits with the error code.

npm Config

builder supports package.jsonconfig properties the same way that npm
does, with slight enhancements in consideration of multiple package.json's
in play.

npm Config Overview

As a refresher, npm utilizes the config field of package.json to make
"per-package" environment variables to scripts tasks. For example, if you
have:

Note that the ability to override via the process environment is unique
to builder and not available in real npm.

Config Notes

Tip - Use String Values

Although config properties can be something like:

"config": {
"enabled":true
}

We strongly recommend that you always set strings like:

"config": {
"enabled":"true"
}

And deal just with string values in your tasks, and files. The reasoning here
is that when overriding values from the command line, the values will always
be strings, which has a potential for messy, hard-to-diagnose bugs if the
overridden value is not also a string.

npmrc Configuration

npm has additional functionality for config values that are not
presently supported, such as issuing commands like
npm config set <pkg-name>:my_name Bill that store values in ~/.npmrc and
then override the package.json values at execution time. We may extend
support for this as well, but not at the present.

Command Line Environment Variables

npm does not support overriding config environment variables from the
actual environment. So doing something in our original example like:

$ npm_package_config_my_name=George npm run get-name
Hello, Bob.

In fact, npm will refuse to even add environment variables starting with
npm_package_config to the npm run environment. E.g.

By contrast, builderdoes pass through environment variables already
existing on the command line, and moreover those overrides takes precedence over
the root and archetype package.json values. Those same examples with builder
show that the environment variables do make it through:

Dependencies and dev dependencies for all of the archetype script tasks.

Configuration files for all script tasks.

In most cases, you won't need to override anything. But, if you do, pick the
most granular scripts command in the archetype you need to override and
define just that in your project's package.jsonscript section. Copy
any configuration files that you need to tweak and re-define the command.

Task Resolution

The easiest bet is to just have one archetype per project. But, multiple are
supported. In terms of scripts tasks, we end up with the following example:

and then in your <root>/package.json using the real lifecycle task name.

"test":"builder run npm:test"

Creating an Archetype

Moving common tasks into an archetype is fairly straightforward and requires
just a few tweaks to the paths defined in configuration and scripts in order
to work correctly.

Initializing a Project

An archetype is simply a standard npm module with a valid package.json. To set
up a new archetype from scratch, make a directory for your new archetype,
initialize npm and link it for ease of development.

$ cd path/to/new/archetype
$ npm init
$ npm link

From your consuming project, you can now link to the archetype directly for ease
of development after including it in your dependencies and creating a
.builderrc as outlined above in configuration.

$ cd path/to/consuming/project
$ npm link new-archetype-name

Managing the dev Archetype

Because builder archetypes are included as simple npm modules, two separate
npm modules are required for archetypes: one for normal dependencies and one for
dev dependencies. Whereas in a non-builder-archetype project you'd specify dev
dependencies in devDependencies, with builder all dev dependencies must be
regular dependencies on a separate dev npm module.

builder is designed so that when defining which archetypes to use in a
consuming project's .builderrc, builder will look for two modules, one named
appropriately in dependencies (ex: my-archetype) and one in
devDependencies but with -dev appended to the name (ex: my-archetype-dev).

To help with managing these while building a builder archetype, install
builder-support
to create and manage a dev/ directory within your archetype project with it's
own package.json which can be published as a separate npm module.
builder-support will not only create a dev/package.json with an appropriate
package name, but will also keep all the other information from your archetype's
primary package.json up to date as well as keep README.md and .gitignore
in parity for hosting the project as a separate npm module.

After this, Node then checks for NODE_PATH for additional paths to search.
This presents a potentially awkward pattern when combined with npm
deduplication / flattening for say a file like:
<root>/node_modules/<archetype>/config/my-config.js that requires
lib-name@right-version as follows:

The Module Pattern

To remedy this situation, we encourage a very simple pattern to have Node.js
require's start from the dev archetype when appropriate by adding a one-line
file to the dev archetype: <archetype-dev>/require.js

// Contents of <archetype-dev>/require.jsmodule.exports= require;

By exporting the require from the dev archetype, the resolution starts in the
dev archetype and thus ensures the dev archetype "wins" for the archetype tasks.
Thus in any archetype files that do a require, simply switch to:

And the dependency from the dev archetype is guaranteed to "win" no matter what
happens with actual module layout from npm installation.

Note that because a file from within the normal <archetype> will naturally
search <archetype>/node_modules before hitting <root>/node_modules you do
not need to use this require pattern for normal archetype dependencies in
archetype Node.js files.

Node.js files in the normal production archetype do not need a
<archetype>/require.js file akin to the dev archetype because
<archetype>/node_modules is already at the top of the require search path.
However, some projects may wish to have an archetype control and provide
application dependencies and dev dependencies, which we discuss in the
next section

ES.next Imports and The Module Pattern

The module pattern works great for any require()-based CommonJS code.
Unfortunately, when using babel and ES.next imports like:

import_from"lodash";

The module pattern is not available because the actual require("lodash")
statement spit out during transpilation is not directly accessible to the
developer.

Fortunately (and unsurprisingly) we have a babel plugin to enable the module
pattern in ES.next code: babel-plugin-replace-require.
The plugin can easily be configured with tokens to insert dev archetypes in
requires produced by babel transpilation. For example, say we wanted to get
lodash from above from our dev archetype, we would configure a .babelrc
like:

Webpack and Module Pattern

An analogous situation occurs for frontend JS code in the production archetype,
but with a different solution. The underlying issue is that Webpack cannot
ingest:

// src/foo.jsvar mod =require("<archetype-dev>/require")("lib-name");

like Node.js can, so we need a little help in the form of a loader.

Note: Previous incarnations of this documentation suggested mutating
Webpack code (resolve.root) and loader (resolveLoader.root) configurations.
We no longer suggest this as the loader pattern in this section is much more
precise and doesn't lead to potential prod-vs-dev ambiguities in module use.

Turning to the above example, we can use the
webpack-alternate-require-loader to
rewrite CommonJS forms of the module pattern into fully-resolved paths on disk
that work for webpack.

With this configuration, Webpack will parse our above code sample and actually
perform the resolution of lib-name using the require provided in
<archetype-dev>/require producing ultimate code like:

// lib/foo.jsvar mod =require("/RESOLVED/PATH/TO/lib-name");

This essentially converts a runtime lookup of the require starting from
the dev archetype to a build time lookup performed by webpack.

Conveniently, this plugin also works with code produced by
babel-plugin-replace-require when configured as specified in the previous
section.

Shared Node / Frontend Code: The best part of this plugin is that if you
have shared code between Node.js and the frontend, you can have the exact same
code work in both places -- unparsed for Node.js and processed via Webpack for
the frontend.

Application vs. Archetype Dependencies

Out of the box builder does not manage application dependencies, instead
managing dependencies only for builder workflows and tasks, e.g. things
starting with the builder command.

Most notably, this means that if your application code includes a dependency
like lodash:

Moving scripts and Config Files

All scripts defined in archetypes will be run from the root of the project
consuming the archetype. This means you have to change all paths in your scripts
to reference their new location within the archetype.

An example script and config you may be moving to an archetype would look like:

Any paths that reference files expected in the consuming app (in this example
test/server/spec) do not need to change.

Updating Path and Module References in Config Files

Any JavaScript files run from within an archetype (such as config files) require
a few changes related to paths now that the files are being run from within
an npm module. This includes all require() calls referencing npm modules and
all paths to files that aren't relative.

All non-relative paths to files and npm modules need to be full paths, even ones
not in the archetype directory. For files expected to be in the consuming
project, this can be achieved by prepending process.cwd() to all paths. For
npm modules, full paths can be achieved by using
require.resolve().

Tips, Tricks, & Notes

PATH, NODE_PATH Resolution

Builder uses some magic to enhance PATH and NODE_PATH to look in the
installed modules of builder archetypes and in the root of your project (per
normal). We mutate both of these environment variables to resolve in the
following order:

The order of resolution doesn't often come up, but can sometimes be a factor
in diagnosing archetype issues and script / file paths, especially when using
npm v3.

Environment Variables

Builder clones the entire environment object before mutating it for further
execution of tasks. On Mac/Linux, this has no real change of behavior of how
the execution environment works. However, on Windows, there are some subtle
issues with the fact that Windows has a case-insensitive environment variable
model wherein you can set PATH in a node process, but internally this is
transformed to set Path. Builder specifically handles PATH correctly across
platforms for it's own specific mutation.

However, if your tasks rely on the Windows coercion of case-insensitivity of
environment variables, you may run into some idiosyncratic problems with tasks.

Alternative to npm link

In some cases, npm link can interfere with the order of resolution. If you
run into resolution problems, you can develop locally with the
following in your consuming project's package.json as an alternative to npm link:

Project Root

The enhancements to NODE_PATH that builder performs can throw tools /
libraries for a loop. Generally speaking, we recommend using
require.resolve("LIBRARY_OR_REQUIRE_PATH") to get the appropriate installed
file path to a dependency.

This comes up in situations including:

Webpack loaders

Karma included files

The other thing that comes up in our Archetype configuration file is the
general requirement that builder is running from the project root, not
relative to an archetype. However, some libraries / tools will interpret
"./" as relative to the configuration file which may be in an archetype.

So, for these instances and instances where you typically use __dirname,
an archetype may need to use process.cwd() and be constrained to only
ever running from the project root. Some scenarios where the process.cwd()
path base is necessary include:

Webpack entry points, aliases

Karma included files (that cannot be require.resolve-ed)

Avoid npm Lifecycle Commands

in your archetype scripts. This is due to the fact that the archetype
package.json files are themselves consumed by npm for publishing (which
can lead to tasks executing for the archetype instead of the project using
the archetype) and potentially lead to awkward recursive composed task
scenarios.

Instead, we recommend adding an npm:<task> prefix to your tasks to identify
them as usable in root projects for real npm lifecycle tasks.

Other Process Execution

The execution of tasks generally must originate from Builder, because of all
of the environment enhancements it adds. So, for things that themselves exec
or spawn processes, like concurrently, this can be a problem. Typically, you
will need to have the actual command line processes invoked by Builder.

I Give Up. How Do I Abandon Builder?

Builder is designed to be as close to vanilla npm as possible. So, if for
example you were using the builder-react-component archetype with a project
package.json like:

and decided to no longer use Builder, here is a rough set of steps to unpack
the archetype into your project and remove all Builder dependencies:

Copy all ARCHETYPE/package.json:dependencies to your
PROJECT/package.json:dependencies (e.g., from builder-react-component).
You do not need to copy over ARCHETYPE/package.json:devDependencies.

Copy all ARCHETYPE/package.json:scripts to your
PROJECT/package.json:scripts that do not begin with the builder: prefix.
Remove the npm: prefix from any scripts tasks and note that you may have
to manually resolve tasks of the same name within the archetype and also with
your project.

Copy all ARCHETYPE-dev/package.json:dependencies to your
PROJECT/package.json:devDependencies
(e.g., from builder-react-component-dev)

Copy all configuration files used in your ARCHETYPE into the root project.
For example, for builder-react-component you would need to copy the
builder-react-component/config directory to PROJECT/config (or a renamed
directory).

Replace all instances of require("<archetype-dev>/require") and
require("<archetype>/require") with require in configuration / other
Node.js files from the archetype.

Review all of the combined scripts tasks and:

resolve duplicate task names

revise configuration file paths for the moved files

replace instances of builder run <task> with npm run <task>

for builder concurrent <task1> <task2> tasks, first install the
concurrently package and then rewrite to:
concurrent 'npm run <task1>' 'npm run <task2>'

... and (with assuredly a few minor hiccups) that's about it! You are
Builder-free and back to a normal npm-controlled project.