Makefile Optimization: $(shell) and := go Together

$(shell) explained$(shell) is GNU Make's equivalent of the back tick (`) operator in the shell. $(shell) executes a command, flattens the result (i.e. turns all line endings into spaces) and returns the resulting string.

For example, if you want to get the output of the date command into a variable called NOW you'd write

NOW = $(shell date)

Or if you want to count the number of files in the current directory and get that number into FILE_COUNT do

FILE_COUNT = $(shell ls | wc -l )

Note that since $(shell) flattens output you can get the names of all the files in the current directory into a macro with:

FILES = $(shell ls)

The newline between files is replaced with a single space making FILES a space-separated list of filenames.

You'll commonly see in Makefiles an execution of the pwd command to get the current working directory into a variable (in this case CWD):

CWD = $(shell pwd)

It's this command we'll take a look at later on when considering how to optimize an example Makefile that wastes time getting the working directory over and over again.

The difference between = and := — 99% of the time you'll see macro definitions in Makefiles that use the = form:

In the example above macros FOO, BAR and FOOBAR are “recursively expanded” macros. That means that when we ask for the value of a macro any macros that it references are expanded at that point. For example, if we ask for the value of $(FOOBAR) it will get the value of $(FOO) and $(BAR) put them together with the space in between and return foo bar. Expansion through as many levels of macros as necessary is done only when the variable is used.

In the Makefile above this has the interesting side effect that FOOBARappears to have two different values. Running it prints out:

foo fooey barney
bar fooey barney

The value of FOOBAR is used to define the list of prerequisites to the all rule and is expanded as foo bar, the same thing happens for the next rule which defines rules for foo and bar.

But when the rules are run, the value of FOOBAR as used in the echo comes out as fooey barney. (You can verify that the value of FOOBAR was foo bar when the rules were defined by looking at the value of [email protected] target being built---when the rules are run).

There are two cases to remember:

When a rule is being defined in a Makefile, macros will evaluate to their value at that point in the Makefile.

Macros used in rule bodies (i.e. in the commands) have their final value: whatever value the macro had at the end of the Makefile.

If we change the definition of FOOBAR to use a := instead of = running the Makefile gives a very different result.

foo foo bar
bar foo bar

Now FOOBAR has the same value everywhere. That because := forces the right-hand side of the definition to be expanded at that moment. Rather than storing $(FOO) $(BAR) as the definition of FOOBAR, GNU Make stores the expansion of $(FOO) $(BAR) which at that point is foo bar. The fact that FOO and BAR are redefined later in the Makefile is irrelevant, FOOBAR has already been expanded and set to a fixed string. GNU Make refers to variables defined in this way as “simply expanded.”

Once a macro has become simply expanded it remains that way unless it is redefined using the = operator. This means that if you append to a simply expanded macro the text being appended is expanded before the append.

results in FOOBAR being foo bar baz. If = had been used instead of := then when $(BAZ) was appended it would not have been expanded and the resulting FOOBAR would have been foo baz bazzy.Take a look at this example Makefile.

It gets the current working directory into CWD, defines a source and object directory as sub-directories of the CWD, defines a set of objects (foo.o, bar.o and baz.o) to be built in the OBJ_DIR, sets up a pattern rule showing how to build a .o from a .c and finally states that by default the Makefile should build all the objects and print out a list of those that were out of date ($? is the list of prerequisites of a rule that were out of date) and a full list of objects.

You might be surprised to learn that this Makefile ends up making eight shell invocations just to get the CWD value. Imagine how many times GNU Make would make costly calls to the shell in a real Makefile with hundreds or thousands of objects!

There are so many calls to $(shell) because the Makefile uses recursively expanded macros: i.e. macros whose value is determined when the macro is used and not at definition time. OBJS references OBJ_DIR three times which references CWD each time; every time OBJS is referenced there are three calls to $(shell pwd). Any other reference to SRC_DIR or OBJ_DIR (e.g. the pattern rule definition) results in another $(shell pwd).

But there's a quick fix for this. Just change the definition of CWD to simply expanded by inserting a : to turn = into :=. Since the working directory doesn't change during the Make we can safely get it once:

CWD := $(shell pwd)

Now there's a single call out to the shell to get the working directory. In a real Makefile this could be a large time saver.

Since it can be hard to follow through a Makefile to see everywhere a macro is used there's a simple trick that will cause Make to print out the exact line at which a macro is expanded. If we insert $(warning Call to shell) in the definition of CWD so that its definition is

About the author

John Graham-Cumming is Co-Founder at Electric Cloud, Inc . Prior to joining Electric Cloud, John was a Venture Consultant with Accel Partners, VP of Internet Technology at Interwoven, Inc. (IWOV), VP of Engineering at Scriptics Corporation (acquired by Interwoven), and Chief Architect at Optimal Networks, Inc. John holds BA and MA degrees in Mathematics and Computation and a Doctorate in Computer Security from Oxford University. John is the creator of the highly acclaimed open source POPFile project. He also holds two patents in network analysis and has others pending.