Parallel Execution

Make can build dependencies in parallel sub-processes, via its --jobs
flag (or its -j abbreviation) which specifies the number of sub-processes to use e.g.

$ make --jobs 4 results.txt

If we have independent dependencies then these can be built at the
same time. For example, abyss.dat and isles.dat are mutually
independent and can both be built at the same time. Likewise for
abyss.png and isles.png. If you’ve got a bunch of independent
branches in your analysis, this can greatly speed up your build
process.

Different Types of Assignment

Some Makefiles may contain := instead of =. Your Makefile may
behave differently depending upon which you use and how you use it:

A variable defined using = is a recursively expanded
variable. Its value is calculated only when its value is
requested. If the value assigned to the variable itself contains
variables (e.g. A = $(B)) then these variables’ values are only
calculated when the variable’s value is requested (e.g. the value of
B is only calculated when the value of A is requested via
$(A). This can be termed lazy setting.

A variable defined using := is a simply expanded variable. Its
value is calculated when it is declared. If the value assigned to
the variable contains variables (e.g. A = $(B)) then these
variables’ values are also calculated when the variable is declared
(e.g. the value of B is calculated when A is assigned
above). This can be termed immediate setting.

Make and Version Control

Imagine that we manage our Makefiles using a version control
system such as Git.

Let’s say we’d like to run the workflow developed in this lesson
for three different word counting scripts, in order to compare their
speed (e.g. wordcount.py, wordcount2.py, wordcount3.py).

To do this we could edit config.mk each time by replacing
COUNT_SRC=wordcount.py with COUNT_SRC=wordcount2.py or
COUNT_SRC=wordcount3.py,
but this would be detected as a change by the version control system.
This is a minor configuration change, rather than a change to the
workflow, and so we probably would rather avoid committing this change
to our repository each time we decide to test a different counting script.

An alternative is to leave config.mk untouched, by overwriting the value
of COUNT_SRC at the command line instead:

$ make variables COUNT_SRC=wordcount2.py

The configuration file then simply contains the default values for the
workflow, and by overwriting the defaults at the command line you can
maintain a neater and more meaningful version control history.

Make Variables and Shell Variables

Makefiles embed shell scripts within them, as the actions that are
executed to update an object. More complex actions could well include
shell variables. There are several ways in which make variables and
shell variables can be confused and can be in conflict.

Make actually accepts three different syntaxes for variables: $N,
$(NAME), or ${NAME}.

The single character variable names are most commonly used for
automatic variables, and there are many of them. But if you happen
upon a character that isn’t pre-defined as an automatic variable,
make will treat it as a user variable.

The ${NAME} syntax is also used by the unix shell in cases where
there might be ambiguity in interpreting variable names, or for
certain pattern substitution operations. Since there are only
certain situations in which the unix shell requires this syntax,
instead of the more common $NAME, it is not familiar to many users.

Make does variable substitution on actions before they are passed to
the shell for execution. That means that anything that looks like a
variable to make will get replaced with the appropriate value. (In
make, an uninitialized variable has a null value.) To protect a
variable you intend to be interpreted by the shell rather than make,
you need to “quote” the dollar sign by doubling it ($$). (This the
same principle as escaping special characters in the unix shell
using the backslash (\) character.) In
short: make variables have a single dollar sign, shell variables
have a double dollar sign. This applies to anything that looks like
a variable and needs to be interpreted by the shell rather than
make, including awk positional parameters (e.g., awk '{print $$1}'
instead of awk '{print $1}') or accessing environment variables
(e.g., $$HOME).

Detailed Example of Shell Variable Quoting

Say we had the following Makefile (and the .dat files had already
been created):

Notice that make substituted $(BOOKS), as expected, but it also
substituted $book, even though we intended it to be a shell variable.
Moreover, because we didn’t use $(NAME) (or ${NAME}) syntax, make
interpreted it as the single character variable $b (which we haven’t
defined, so it has a null value) followed by the text “ook”.

In order to get the desired behavior, we have to write $$book instead
of $book: