PR/46096: Jarmo Jaakkola: fix many problems with dependencies (PR 49086)
Quite extensive rewrite of the Suff module. Some ripple effects into
Parse and Targ modules too.
Dependency searches in general were made to honor explicit rules so
implicit and explicit sources are no longer applied on targets that
do not invoke a transformation rule.
Archive member dependency search was rewritten. Explicit rules now
work properly and $(.TARGET) is set correctly. POSIX semantics for
lib(member.o) and .s1.a rules are supported.
.SUFFIXES list maintenance was rewritten so that scanning of existing
rules works when suffixes are added and that clearing the suffix list
removes single suffix rules too. Transformation rule nodes are now
mixed with regular nodes so they are available as regular targets too
if needed (especially after the known suffixes are cleared).
The .NULL target was documented in the manual page, especially to
warn against using it when a single suffix rule would work.
A deprecation warning was also added to the manual and make also
warns the user if it encounters .NULL.
Search for suffix rules no longer allows the explicit dependencies
to override the selected transformation rule. A check is made in
the search that the transformation that would be tried does not
already exist in the chain. This prevents getting stuck in an infinite
loop under specific circumstances. Local variables are now set
before node's children are expanded so dynamic sources work in
multi-stage transformations. Make_HandleUse() no longer expands
the added children for transformation nodes, preventing triple
expansion and allowing the Suff module to properly postpone their
expansion until proper values are set for the local variables.
Directory prefix is no longer removed from $(.PREFIX) if the target
is found via directory search.
The last rule defined is now used instead of the first one (POSIX
requirement) in case a rule is defined multiple times. Everything
defined in the first instance is undone, but things added "globally"
are honored. To implement this, each node tracks attribute bits
which have been set by special targets (global) instead of special
sources (local). They also track dependencies that were added by
a rule with commands (local) instead of rule with no commands (global).
New attribute, OP_FROM_SYS_MK is introduced. It is set on all targets
found in system makefiles so that they are not eligible to become
the main target. We cannot just set OP_NOTMAIN because it is one of
the attributes inherited from transformation and .USE rules and would
make any eligible target that uses a built-in inference rule ineligible.
The $(.IMPSRC) local variable now works like in gmake: it is set to
the first prerequisite for explicit rules. For implicit rules it
is still the implied source.
The manual page is improved regarding the fixed features. Test cases
for the fixed problems are added.
Other improvements in the Suff module include:
- better debug messages for transformation rule search (length of
the chain is now visualized by indentation)
- Suff structures are created, destroyed and moved around by a set
of maintenance functions so their reference counts are easier
to track (this also gets rid of a lot of code duplication)
- some unreasonably long functions were split into smaller ones
- many local variables had their names changed to describe their
purpose instead of their type

Don't use emalloc and friends directly, but call them consistently
bmake_malloc and friends. Implement them via macros for the native case
and provide fallback implementations otherwise. Avoid polluting the
namespace by not defining enomem globally. Don't bother to provide
strdup and strndup, they were only used for the estrdup and estrndup
comapt code.
This addresses the presence of emalloc in system libraries on A/UX and
resulted strange issues as reported by Timothy E. Larson.

A rather large rototil in the way the parallel make code schedules jobs.
This gives a considerable speedup in the processing of .WAIT and .ORDER.
Both .WAIT and .ORDER stop both the commands of the node, and its dependant
nodes being built until the LH nodes are complete.
.WAIT only applies to the dependency line on which it appears, whereas
.ORDER applies globally between the two nodes.
In both cases dependant nodes can be built because other targets need them.
make now processes the target list left to right, scheduling child nodes
as they are needed to make other nodes (instead of attempting to generate
a bottom-up dependency graph at the start). This means that 'make -j1'
will tend to build in the same order as a non-parallel make.
Note that:
all: x y
x: a .WAIT b
y: b .WAIT a
does not generate a dependency loop.
But
x: y
.ORDER y x
does (unless something elswhere causes 'y' to be built).

Only do dynamic dependecy expansion once, and follow by a single filename
globbing.
The old behaviour was the perform variable expansion and globbing on the
output of both the variable expansion and globbing. Which allows some very
strange behaviour if, for example, globbed filenames contain $ symbols.
Unconditionally add new nodes from these expansions even if the names are
already children. The .WAIT code needs the order of children preserved.

Output all debug trace output through 'debug_file' defaulting to 'stdout'.
(Almost all the debug output went there, but some went to stderr.)
Split the parsing of -d (debug flags) out into its own routine.
Allow the output filename to be changed by specifying -dF<file> to create
a log file, or -dF+<file> to append to it. <file> may be stdout or stderr.
Also change so that -d-<flags> acts on <flags> locally but doesn't copy
them to MAKEFLAGS so they aren't inherited by child makes.
I'm not 100% happy with the command line syntax for the above, so they are
currently undocumented.

Add some coverity allocation comments, and change the way the allocator
functions work. When they allocate storage that needs to be freed, instead
of setting a boolean, set the pointer to be freed. Plug some more memory
leaks found by inspection.

Dir_MTime did not search for a file using the correct parh; i.e. it was
ignoring suffix-specific path search. So if a node was marked .MADE,
then suffix rules would not be applied to it, and we would look for
the file only in the default path, not the suffix-specific path.
XXX: Now that we looked for the suffix, we can save it in the GNode,
but we don't do this yet.

Simplify build, no functional changes.
Instead of adding MAKE_BOOTSTRAP for hosted environments, i.e., when
you want things simple, instead add MAKE_NATIVE to get those hugely
important features like __RCSID().
It's now possible to build make on some hosts with: cc *.c */*.c

Bring down the number of stat(2) system calls from 682 to 294. This change
adds a .NOSUFF directive that has been applied to targets that have been
already made, and are not supposed to have suffix rules applied to them.

Fix major bug in make(1) ... due to shadowing of the dotLast path used for
the .DOTLAST primitive by a boolean variable with the same name, this whole
mechanism was broken ... it doesn't save much stat calls but it was wrong.
Thanks to Jason Thorpe for the other shadow-variable fixing patches he
made.

Fix a very old and annoying bug: Adding suffixes to sufflist wasn't setting
a refCount of 1 to indicate its presence on sufflist. Hence suffixes were
being removed by Suff_EndTransform() and being freed before all the
referential links were removed. This resulted in a malloc warning on 1.5.x:
make in free(): warning: chunk is already free.
This could be reproduced with the following simple Makefile passed to
"make -r":
=====
.SUFFIXES: .l .c
.l.c:
foo:

Readd optimization last night. Problems earlier were partially due to the
arguments names on one function being swapped (by a previous author).
Do not do any duplicate suppression when a source list is created. Instead:
* OP_MADE protects against trying to make the source multiple times.
* A new OP_MARK flag is introduced to suppress duplicates while expanding
the .ALLSRC variable and .USE targets.
This turns the O(n^2) insertion into O(n) in most cases.
This is tested with a `make build' and some special test cases.

Rework how :: dependencies are handled.
Build a list of `cohorts' as before, but do *not* link each one into all the
parent nodes; instead, copy the `cohort' lists into the stream of targets to
be built inside Make_ExpandUse(). Also do the attribute propagation as a
separate pass after parsing.
This eliminates several O(n^2) algorithms.

Avoid using Lst_Member() in SuffExpandChildren(), by avoiding Lst_ForEach() and
passing in the LstNode of the child being inspected. Shaves off another few %,
particularly when there are long child lists containing $ expansions (e.g. in
libc).

If the target name is shorter than the suffix,
the previous garbage bytes may be read.
Example: "make n" --- target = n, suffix = .ln
Changing interface of SuffSuffIsSuffix() is required to fix this bug.

- Target searching addition:
Make used to only use the search path for nodes that were pure
sources (not targets of other sources). This has been corrected
and now gnu-autoconf generated Makefiles work in directories other
than the source one.
- Suffix transformation rescanning:
Suffix transformations (.c.o:; cc ...) were only recognized in
the past when both suffixes were members of the suffix list.
Thus a sequence like:
.z.b:
echo ${.TARGET}
.SUFFIXES: .z
would cause .z.b: to be inserted as a regular target (and the main
target in this case). Other make programs always add rules that
start with a period in the transformation list and never consider
them as targets. We cannot do that (consider .depend files) so we
resort to scanning the list of the current targets every time a
suffix gets added, and we mutate existing targets that are now
valid transformation rules into transformation rules. If the
transformed target was also the main target, we set the main target
to be the next target in the targets list.

Minor:
- ${.PREFIX} should never contain a full pathname
- Fixed gcc -Wall warnings
Major:
- compatMake is now FALSE. This means that we are now running in
full pmake mode:
* rules on dependency lines can be executed in parallel and or
out of sequence:
foo: bar baz
can fire the rule for baz before the rule for bar is fired.
To enforce bar to be fired before baz, another rule needs to be
added. [bar: baz]
* adjacent shell commands in a target are now executed by a single
invocation of the shell, not one invocation of the shell per line
(compatMake can be turned off using the -B flag)
- The -j flag now works... I.e. make -j 4 will fork up to four jobs in
parallel when it can. The target name is printed before each burst
of output caused by the target execution as '--- target ---', when j > 1
- I have changed all the Makefiles so that they work with make -j N, and
I have tested the whole netbsd by:
'make -j 4 cleandir; make -j 4 depend; make -j 4; make -j 4 install'
- I have not compiled or tested this version of make with -DREMOTE.

Fixes from Christos Zoulas:
The following two patches fix a couple of problems with make(1)
1. Null Suffixes were not being copied, but they were being free'd
This caused rules of the form:
.c:
${CC} ...
to access invalid memory and potentially core dump..
[That was always broken; I did not break that one :-)]
2. My recent fixes to parse ${VAR:%.foo=%.bar} handled the null string
case incorrectly (${VAR:=.c} was broken).