=head1IntroductionB<Cons>isasystemforconstructing,primarily,software,butisquitedifferentfromprevioussoftwareconstructionsystems.Conswasdesignedfromthegrounduptodealeasilywiththeconstructionofsoftwarespreadovermultiplesourcedirectories.Consmakesiteasytocreatebuildscriptsthataresimple,understandableandmaintainable.Consensuresthatcomplexsoftwareiseasilyandaccuratelyreproducible.Consusesanumberoftechniquestoaccomplishallofthis.ConstructionscriptsarejustPerlscripts,makingthembotheasytocomprehendandveryflexible.Globalscopingofvariablesisreplacedwithanimport/exportmechanismforsharinginformationbetweenscripts,significantlyimprovingthereadabilityandmaintainabilityofeachscript.B<Constructionenvironments>areintroduced:thesearePerlobjectsthatcapturetheinformationrequiredforcontrollingthebuildprocess.Multipleenvironmentsareusedwhendifferentsemanticsarerequiredforgeneratingproductsinthebuildtree.Consimplementsautomaticdependencyanalysisandusesthistogloballysequencetheentirebuild.Variantbuildsareeasilyproducedfromasinglesourcetree.Intelligentbuildsubsettingispossible,whenworkingonlocalizedchanges.Overridescanbesetuptoeasilyoverridebuildinstructionswithoutmodifyinganyscripts.MD5cryptographicB<signatures>areassociatedwithderivedfiles,andareusedtoaccuratelydeterminewhetheragivenfileneedstoberebuilt.Whileofferingalloftheabove,andmore,Consremainssimpleandeasytouse.Thiswill,hopefully,becomeclearasyoureadtheremainderofthisdocument.=head2AutomaticglobalbuildsequencingBecauseConsdoesfullandaccuratedependencyanalysis,anddoesthisglobally,fortheentirebuild,ConsisabletousethisinformationtotakefullcontroloftheB<sequencing>ofthebuild.Thissequencingisevidentintheaboveexamples,andisequivalenttowhatyouwouldexpectformake,givenafullsetofdependencies.WithCons,thisextendstriviallytolarger,multi-directorybuilds.Asaresult,allofthecomplexityinvolvedinmakingsurethatabuildisorganizedcorrectly--includingmulti-passhierarchicalbuilds--iseliminated.We'll discuss this further in the nextsections.=head1 A Model for sharing files=head2 Some simple conventionsIn any complex software system, a method for sharing build products needs tobe established. We propose a simple set of conventions which are trivial toimplement with Cons, but very effective.The basic rule is to require that all build products which need to be sharedbetween directories are shared via an intermediate directory. We havetypically called this F<export>, and, in a C environment, providedconventional sub-directories of this directory, such as F<include>, F<lib>,F<bin>, etc.These directories are defined by the top-level F<Construct> file. A simpleF<Construct> file for a B<Hello, World!> application, organized usingmultiple directories, might look like this: # Construct file for Hello, World! # Where to put all our shared products. $EXPORT = '#export'; Export qw( CONS INCLUDE LIB BIN ); # Standard directories for sharing products. $INCLUDE = "$EXPORT/include"; $LIB = "$EXPORT/lib"; $BIN = "$EXPORT/bin"; # A standard construction environment. $CONS = new cons ( CPPPATH => $INCLUDE, # Include path for C Compilations LIBPATH => $LIB, # Library path for linking programs LIBS => '-lworld', # List of standard libraries ); Build qw( hello/Conscript world/Conscript );The F<world> directory'sF<Conscript>filelookslikethis:#ConscriptfilefordirectoryworldImportqw(CONSINCLUDELIB);#InstalltheproductsofthisdirectoryInstall$CONS$LIB,'libworld.a';Install$CONS$INCLUDE,'world.h';#InternalproductsLibrary$CONS'libworld.a','world.c';andtheF<hello>directory's F<Conscript> file looks like this: # Conscript file for directory hello Import qw( CONS BIN ); # Exported products Install $CONS $BIN, 'hello'; # Internal products Program $CONS 'hello', 'hello.c';To construct a B<Hello, World!> program with this directory structure, go tothe top-level directory, and invoke C<cons> with the appropriatearguments. In the following example, we tell Cons to build the directoryF<export>. To build a directory, Cons recursively builds all known productswithin that directory (only if they need rebuilding, of course). If any ofthose products depend upon other products in other directories, then thosewill be built, too. % cons export Install world/world.h as export/include/world.h cc -Iexport/include -c hello/hello.c -o hello/hello.o cc -Iexport/include -c world/world.c -o world/world.o ar r world/libworld.a world/world.o ar: creating world/libworld.a ranlib world/libworld.a Install world/libworld.a as export/lib/libworld.a cc -o hello/hello hello/hello.o -Lexport/lib -lworld Install hello/hello as export/bin/hello=head2 Clean, understandable, location-independent scriptsYou'llnotethatthetwoF<Conscript>filesareverycleanandto-the-point.Theysimplyspecifyproductsofthedirectoryandhowtobuildthoseproducts.Thebuildinstructionsareminimal:theyspecifywhichconstructionenvironmenttouse,thenameoftheproduct,andthenameoftheinputs.Notealsothatthescriptsarelocation-independent:ifyouwishtoreorganizeyoursourcetree,youarefreetodoso:youonlyhavetochangetheF<Construct>file(inthisexample),tospecifythenewlocationsoftheF<Conscript>files.Theuseofanexporttreemakesthisgoaleasy.Note,too,howConstakescareoflittledetailsforyou.AlltheF<export>directories,forexample,weremadeautomatically.Andtheinstalledfileswerereallyhard-linkedintotherespectiveexportdirectories,tosavespaceandtime.Thisattentiontodetailsavesconsiderablework,andmakesiteveneasiertoproducesimple,maintainablescripts.=head1SignaturesConsusesfileB<signatures>todecideifaderivedfileisout-of-dateandneedsrebuilding.Inessence,ifthecontentsofafilechange,orthemannerinwhichthefileisbuiltchanges,thefile's signaturechanges as well. This allows Cons to decide with certainty when a fileneeds rebuilding, because Cons can detect, quickly and reliably, whetherany of its dependency files have been changed.=head2 MD5 content and build signaturesCons uses the B<MD5> (B<Message Digest 5>) algorithm to compute filesignatures. The MD5 algorithm computes a strong cryptographic checksumfor any given input string. Cons can, based on configuration, use twodifferent MD5 signatures for a given file:The B<content signature> of a file is an MD5 checksum of the file'scontents.Consequently,whenthecontentsofafilechange,itscontentsignaturechangesaswell.TheB<buildsignature>ofafileisacombinedMD5checksumof:=over4thesignaturesofalltheinputfilesusedtobuildthefilethesignaturesofalldependencyfilesdiscoveredbysourcescanners(forexample,C<.h>files)thesignaturesofalldependencyfilesspecifiedexplicitlyviatheC<Depends>method)thecommand-linestringusedtobuildthefile=backThebuildsignatureis,ineffect,adigestofallthedependencyinformationforthespecifiedfile.Consequently,afile's buildsignature changes whenever any part of its dependency informationchanges: a new file is added, the contents of a file on which it dependschange, there'sachangetothecommandlineusedtobuildthefile(oranyofitsdependencyfiles),etc.Forexample,intheprevioussection,thebuildsignatureoftheF<world.o>filewillinclude:=over4thesignatureoftheF<world.c>filethesignaturesofanyheaderfilesthatConsdetectsareincluded,directlyorindirectly,byF<world.c>thetextoftheactualcommandlinewasusedtogenerateF<world.o>=backSimilarly,thebuildsignatureoftheF<libworld.a>filewillincludeallthesignaturesofitsconstituents(andhence,transitively,thesignaturesofB<their>constituents),aswellasthecommandlinethatcreatedthefile.NotethatthereisnoneedforaderivedfiletodependuponanyparticularF<Construct>orF<Conscript>file.Ifchangestothesefilesaffectafile,thenthiswillbeautomaticallyreflectedinitsbuildsignature,sincerelevantpartsofthecommandlineareincludedinthesignature.UnrelatedF<Construct>orF<Conscript>changeswillhavenoeffect.=head2Storingsignaturesin.consignfilesBeforeConsexits,itstoresthecalculatedsignaturesforallofthefilesitbuiltorexaminedinF<.consign>files,oneperdirectory.Consusesthisstoredinformationonlaterinvocationstodecideifderivedfilesneedtoberebuilt.Afterthepreviousexamplewascompiled,theF<.consign>fileintheF<build/peach/world>directorylookedlikethis:world.h:985533370-d181712f2fdc07c1f05d97b16bfad904world.o:9855333722a0f71e0766927c0532977b0d2158981world.c:985533370-c712f77189307907f4189b5a7ab62ff3libworld.a:98553337469e568fc5241d7d25be86d581e1fb6aaAfterthefilenameandcolon,thefirstnumberisatimestampofthefile's modification time (on UNIX systems, this is typically the numberof seconds since January 1st, 1970). The second value is the buildsignature of the file (or ``-'' in the case of files with no buildsignature--that is, source files). The third value, if any, is thecontent signature of the file.=head2 Using build signatures to decide when to rebuild filesWhen Cons is deciding whether to build or rebuild a derived file, itfirst computes the file'scurrentbuildsignature.Ifthefiledoesn'texist, it must obviously be built.If, however, the file already exists, Cons next compares themodification timestamp of the file against the timestamp value inthe F<.consign> file. If the timestamps match, Cons compares thenewly-computed build signature against the build signature in theF<.consign> file. If the timestamps do not match or the buildsignatures do not match, the derived file is rebuilt.After the file is built or rebuilt, Cons arranges to store thenewly-computed build signature in the F<.consign> file when it exits.=head2 Signature exampleThe use of these signatures is an extremely simple, efficient, andeffective method of improving--dramatically--the reproducibility of asystem.We'lldemonstratethiswithasimpleexample:#Simple"Hello, World!"Constructfile$CFLAGS='-g'if$ARG{DEBUG}eq'on';$CONS=newcons(CFLAGS=>$CFLAGS);Program$CONS'hello','hello.c';NoticehowConsrecompilesattheappropriatetimes:% cons hellocc-chello.c-ohello.occ-ohellohello.o% cons hellocons:"hello"isup-to-date.% cons DEBUG=on hellocc-g-chello.c-ohello.occ-ohellohello.o% cons DEBUG=on hellocons:"hello"isup-to-date.% cons hellocc-chello.c-ohello.occ-ohellohello.o=head2Source-filesignatureconfigurationConsprovidesaC<SourceSignature>methodthatallowsyoutoconfigurehowthesignatureshouldbecalculatedforanysourcefilewhenitssignatureisbeingusedtodecideifadependentfileisup-to-date.TheargumentstotheC<SourceSignature>methodconsistofoneormorepairsofstrings:SourceSignature'auto/*.c'=>'content','*'=>'stored-content';Thefirststringineachpairisapatterntomatchagainstderivedfilepathnames.Thepatternisafile-globbingpattern,notaPerlregularexpression;thepattern<*.l>willmatchallLexsourcefiles.TheC<*>wildcardwillmatchacrossdirectoryseparators;thepatternC<foo/*.c>would match all C source files in any subdirectory underneath the C<foo>subdirectory.The second string in each pair contains one of the following keywords tospecify how signatures should be calculated for source files that matchthe pattern. The available keywords are:=over 4=item contentUse the content signature of the source file when calculating signaturesof files that depend on it. This guarantees correct calculation of thefile's signature for all builds, by telling Cons to read the contents ofa source file to calculate its content signature each time it is run.=item stored-contentUse the source file's content signature as stored in the F<.consign>file, provided the file's timestamp matches the cached timestamp valuein the F<.consign> file. This optimizes performance, with the slightrisk of an incorrect build if a source file's contents have been changedso quickly after its previous update that the timestamp still matchesthe stored timestamp in the F<.consign> file even though the contentshave changed.=backThe Cons default behavior of always calculating a source file'ssignature from the file's contents is equivalent to specifying: SourceSignature '*' => 'content';The C<*> will match all source files. The C<content> keywordspecifies that Cons will read the contents of a source file to calculateits signature each time it is run.A useful global performance optimization is: SourceSignature '*' => 'stored-content';This specifies that Cons will use pre-computed content signaturesfrom F<.consign> files, when available, rather than re-calculating asignature from the the source file's contents each time Cons is run. Inpractice, this is safe for most build situations, and only a problemwhen source files are changed automatically (by scripts, for example).The Cons default, however, errs on the side of guaranteeing a correctbuild in all situations.Cons tries to match source file path names against the patterns in theorder they are specified in the C<SourceSignature> arguments: SourceSignature '/usr/repository/objects/*' => 'stored-content', '/usr/repository/*' => 'content', '*.y' => 'content', '*' => 'stored-content';In this example, all source files under the F</usr/repository/objects>directory will use F<.consign> file content signatures, source filesanywhere else underneath F</usr/repository> will not use F<.consign>signature values, all Yacc source files (C<*.y>) anywhere else will notuse F<.consign> signature values, and any other source file will useF<.consign> signature values.=head2 Derived-file signature configurationCons provides a C<SIGNATURE> construction variable that allows you toconfigure how signatures are calculated for any derived file when itssignature is being used to decide if a dependent file is up-to-date.The value of the C<SIGNATURE> construction variable is a Perl arrayreference that holds one or more pairs of strings, like the arguments tothe C<SourceSignature> method.The first string in each pair is a pattern to match against derived filepath names. The pattern is a file-globbing pattern, not a Perl regularexpression; the pattern `*.obj' will match all (Win32) object files.The C<*> wildcard will match across directory separators; the pattern`foo/*.a' would match all (UNIX) library archives in any subdirectoryunderneath the foo subdirectory.The second string in each pair contains one of the following keywordsto specify how signatures should be calculated for derived files thatmatch the pattern. The available keywords are the same as for theC<SourceSignature> method, with an additional keyword:=over 4=item buildUse the build signature of the derived file when calculating signaturesof files that depend on it. This guarantees correct builds by forcingCons to rebuild any and all files that depend on the derived file.=item contentUse the content signature of the derived file when calculating signaturesof files that depend on it. This guarantees correct calculation of thefile's signature for all builds, by telling Cons to read the contents ofa derived file to calculate its content signature each time it is run.=item stored-contentUse the derived file's content signature as stored in the F<.consign>file, provided the file's timestamp matches the cached timestamp valuein the F<.consign> file. This optimizes performance, with the slightrisk of an incorrect build if a derived file's contents have beenchanged so quickly after a Cons build that the file's timestamp stillmatches the stored timestamp in the F<.consign> file.=backThe Cons default behavior (as previously described) for usingderived-file signatures is equivalent to: $env = new cons(SIGNATURE => ['*' => 'build']);The C<*> will match all derived files. The C<build> keyword specifiesthat all derived files' build signatures will be used when calculatingwhether a dependent file is up-to-date.A useful alternative default C<SIGNATURE> configuration for many sites: $env = new cons(SIGNATURE => ['*' => 'content']);In this configuration, derived files have their signatures calculatedfrom the file contents. This adds slightly to Cons' workload, but hasthe useful effect of "stopping" further rebuilds if a derived file isrebuilt to exactly the same file contents as before, which usuallyoutweighs the additional computation Cons must perform.For example, changing a comment in a C file and recompiling shouldgenerate the exact same object file (assuming the compiler doesn'tinsert a timestamp in the object file's header). In that case,specifying C<content> or C<stored-content> for the signature calculationwill cause Cons to recognize that the object file did not actuallychange as a result of being rebuilt, and libraries or programs thatinclude the object file will not be rebuilt. When C<build> isspecified, however, Cons will only "know" that the object file wasrebuilt, and proceed to rebuild any additional files that include theobject file.Note that Cons tries to match derived file path names against thepatterns in the order they are specified in the C<SIGNATURE> arrayreference: $env = new cons(SIGNATURE => ['foo/*.o' => 'build', '*.o' => 'content', '*.a' => 'stored-content', '*' => 'content']);In this example, all object files underneath the F<foo> subdirectorywill use build signatures, all other object files (including objectfiles underneath other subdirectories!) will use F<.consign> filecontent signatures, libraries will use F<.consign> file buildsignatures, and all other derived files will use content signatures.=head2 Debugging signature calculationCons provides a C<-S> option that can be used to specify what internalPerl package Cons should use to calculate signatures. The default Consbehavior is equivalent to specifying C<-S md5> on the command line.The only other package (currently) available is an C<md5::debug>package that prints out detailed information about the MD5 signaturecalculations performed by Cons: % cons -S md5::debug hello sig::md5::srcsig(hello.c) => |52d891204c62fe93ecb95281e1571938| sig::md5::collect(52d891204c62fe93ecb95281e1571938) => |fb0660af4002c40461a2f01fbb5ffd03| sig::md5::collect(52d891204c62fe93ecb95281e1571938, fb0660af4002c40461a2f01fbb5ffd03, cc -c %< -o %>) => |f7128da6c3fe3c377dc22ade70647b39| sig::md5::current(|| eq |f7128da6c3fe3c377dc22ade70647b39|) cc -c hello.c -o hello.o sig::md5::collect() => |d41d8cd98f00b204e9800998ecf8427e| sig::md5::collect(f7128da6c3fe3c377dc22ade70647b39, d41d8cd98f00b204e9800998ecf8427e, cc -o %> %< ) => |a0bdce7fd09e0350e7efbbdb043a00b0| sig::md5::current(|| eq |a0bdce7fd09e0350e7efbbdb043a00b0|) cc -o hello, hello.o=head1 Temporary overridesCons provides a very simple mechanism for overriding aspects of a build. Theessence is that you write an override file containing one or moreC<Override> commands, and you specify this on the command line, when you runC<cons>: % cons -o over exportwill build the F<export> directory, with all derived files subject to theoverrides present in the F<over> file. If you leave out the C<-o> option,then everything necessary to remove all overrides will be rebuilt.=head2 Overriding environment variablesThe override file can contain two types of overrides. The first is incomingenvironment variables. These are normally accessible by the F<Construct>file from the C<%ENV> hash variable. These can trivially be overridden inthe override file by setting the appropriate elements of C<%ENV> (thesecould also be overridden in the user's environment, of course).=head2 The Override commandThe second type of override is accomplished with the C<Override> command,which looks like this: Override <regexp>, <var1> => <value1>, <var2> => <value2>, ...;The regular expression I<regexp> is matched against every derived file thatis a candidate for the build. If the derived file matches, then thevariable/value pairs are used to override the values in the constructionenvironment associated with the derived file.Let's suppose that we have a construction environment like this: $CONS = new cons( COPT => '', CDBG => '-g', CFLAGS => '%COPT %CDBG', );Then if we have an override file F<over> containing this command: Override '\.o$', COPT => '-O', CDBG => '';then any C<cons> invocation with C<-o over> that creates F<.o> files viathis environment will cause them to be compiled with C<-O >and no C<-g>. Theoverride could, of course, be restricted to a single directory by theappropriate selection of a regular expression.Here's the original version of the Hello, World! program, built with thisenvironment. Note that Cons rebuilds the appropriate pieces when theoverride is applied or removed: % cons hello cc -g -c hello.c -o hello.o cc -o hello hello.o % cons -o over hello cc -O -c hello.c -o hello.o cc -o hello hello.o % cons -o over hello cons: "hello" is up-to-date. % cons hello cc -g -c hello.c -o hello.o cc -o hello hello.oIt's important that the C<Override> command only be used for temporary,on-the-fly overrides necessary for development because the overrides are notplatform independent and because they rely too much on intimate knowledge ofthe workings of the scripts. For temporary use, however, they are exactlywhat you want.Note that it is still useful to provide, say, the ability to create a fullyoptimized version of a system for production use--from the F<Construct> andF<Conscript> files. This way you can tailor the optimized system to theplatform. Where optimizer trade-offs need to be made (particular files maynot be compiled with full optimization, for example), then these can berecorded for posterity (and reproducibility) directly in the scripts.=head2 The C<Module> methodThe C<Module> method is a combination of the C<Program> and C<Command>methods. Rather than generating an executable program directly, this commandallows you to specify your own command to actually generate a module. Themethod is invoked as follows: Module $env <module name>, <source or object files>, <construction command>;This command is useful in instances where you wish to create, for example,dynamically loaded modules, or statically linked code libraries.=head2 The C<RuleSet> methodThe C<RuleSet> method returns the construction variables for buildingvarious components with one of the rule sets supported by Cons. Thecurrently supported rule sets are:=over 4=item msvcRules for the Microsoft Visual C++ compiler suite.=item unixGeneric rules for most UNIX-like compiler suites.=backOn systems with more than one available compiler suite, this allows youto easily create side-by-side environments for building software withmultiple tools: $msvcenv = new cons(RuleSet("msvc")); $cygnusenv = new cons(RuleSet("unix"));In the future, this could also be extended to other platforms thathave different default rule sets.=head2 The C<DefaultRules> methodThe C<DefaultRules> method sets the default construction variables thatwill be returned by the C<new> method to the specified arguments: DefaultRules(CC => 'gcc', CFLAGS => '', CCCOM => '%CC %CFLAGS %_IFLAGS -c %< -o %>'); $env = new cons(); # $env now contains *only* the CC, CFLAGS, # and CCCOM construction variablesCombined with the C<RuleSet> method, this also provides an easy wayto set explicitly the default build environment to use some supportedtoolset other than the Cons defaults: # use a UNIX-like tool suite (like cygwin) on Win32 DefaultRules(RuleSet('unix')); $env = new cons();Note that the C<DefaultRules> method completely replaces the defaultconstruction environment with the specified arguments, it does notsimply override the existing defaults. To override one or morevariables in a supported C<RuleSet>, append the variables and values: DefaultRules(RuleSet('unix'), CFLAGS => '-O3'); $env1 = new cons(); $env2 = new cons(); # both $env1 and $env2 have 'unix' defaults # with CFLAGS set to '-O3'=head2 The C<SourcePath> methodThe C<SourcePath> mathod returns the real source path name of a file,as opposed to the path name within a build directory. It is invokedas follows: $path = SourcePath <buildpath>;=head2 The C<ConsPath> methodThe C<ConsPath> method returns true if the supplied path is a derivablefile, and returns undef (false) otherwise.It is invoked as follows: $result = ConsPath <path>;=head2 The C<SplitPath> methodThe C<SplitPath> method looks up multiple path names in a string separatedby the default path separator for the operating system (':' on UNIXsystems, ';' on Windows NT), and returns the fully-qualified names.It is invoked as follows: @paths = SplitPath <pathlist>;The C<SplitPath> method will convert names prefixed '#' to theappropriate top-level build name (without the '#') and will convertrelative names to top-level names.=head2 The C<DirPath> methodThe C<DirPath> method returns the build path name(s) of a directory orlist of directories. It is invoked as follows: $cwd = DirPath <paths>;The most common use for the C<DirPath> method is: $cwd = DirPath '.';to fetch the path to the current directory of a subsidiary F<Conscript>file.=head2 The C<FilePath> methodThe C<FilePath> method returns the build path name(s) of a file orlist of files. It is invoked as follows: $file = FilePath <path>;