Summary of Contents

BuildContrib can be used to create a build script for your extension.
It is inspired by the Java ANT build tool developed by the Apache project,
but is targeted specifically at building Foswiki extensions. It is also
used for Foswiki release builds. The advantage of using BuildContrib is that it
dictates a standard structure and build procedure, which makes your extension
easier for you, and others, to build and maintain.

Features:

not just for building code modules, can also be used to package WikiApplications, and even pure documentation packages.

automatically generates an installer script that can help simplify end-user installation.

includes a wizard script for creating a new extension.

supports creating collections of extensions.

Notes

This is a build and packaging module for use by developers, not an install module for end users (though it does build an installer script, among other things).

The module has only been tested on Linux, but should work with Cygwin OK.

Fast Start

If you don't like reading documentation, and just want a fast route to creating a new extension, then:

Install the contrib using the instructions below

cd to the root of your git foswiki/distro checkout

perl core/create_new_extension.plextension_name

Init the new Extension directory as its own git repository. (Don't add extensions into the distro repository!)

then modify the extension files as required (including MANIFEST). Then when you are ready to create archives:

cd to the [extensions_name]/lib/Foswiki/[Plugins|Contrib]/[extension_name] directory.

Development Model

Archaic AddOn modules are supported but should not be used for new extensions.

you have some passing familiarity with build tools such as make,

you are not developing your extension in a 'production' installation (something that is usually a really bad idea), but are instead doing the sensible thing and developing in a separate directory tree

usually - but not always - a git checkout area.

Standard directory structure

BuildContrib is used to build the Foswiki core, as well as most extensions. This
document will focus on its use for building extensions. See the
tools/build.pl file in a git checkout of foswiki/distro for information on building
the core.

Extensions are developed in subdirectories of the foswiki/distro checkout.
For example, BathPlugin will be developed in BathPlugin.
This directory is called the root directory for the extension.

Even though your extension root directory is
within foswiki/distro, it is almost never checked into the foswiki/distro
repository. foswiki/distro is reserved for default components that we ship
with foswik.

The standard directory structure under a root directory mirrors a standard
installation tree. Every plugin has some key files:

lib/

Foswiki/

Plugins/ (or) Contrib/

name.pm - code file for the plugin or contrib, usually derived from EmptyPlugin or EmptyContrib

name/ - directory containing sub-modules used by your plugin/contrib, and for storing your build.pl script and other support files. It is referred to as the module directory

build.pl - build script for this extension

MANIFEST - list of files to be installed

DEPENDENCIES - list of modules this extension depends on

Config.spec - configure setup for this extension

data/

System/

name.txt - your plugin/contrib topic

test/

unit/

name/ - directory containing unit tests for the extension

pub/

System/

name/ - directory where all your images, css, and Javascript files should go

Contribs are held in the lib/Foswiki/Contrib directory instead of lib/Foswiki/Plugins but otherwise in exactly work the same way.

Foswiki configure now requires a perl module in either the Plugins or Contrib directory with the same name as the
extension. This module should have two global variables - $VERSION and $RELEASE which are used to determine the installed version of the extension.
Plugins also need $SHORTDESCRIPTION which shows up on InstalledPlugins and is merged with the System/YourPlugin topic when build is run,

Other directories normally found in a Foswiki installation may also exist under a root directory e.g. tools, bin, templates etc.

While development in a git checkout is strongly recommended, it is also possible to develop in a normal Foswiki install. To do this, simply install the BuildContrib.

Setting up for Development

Environment Variables

Your build script has to know how to find the Foswiki libraries, so it can
pick up the components of the build system. Set FOSWIKI_LIBS (which is a
path, same as PERL5LIB) to point to your lib directory in your development
Foswiki. $FOSWIKI_LIBS is used to extend @INC _for the duration of the build
only_, so it won't mask problems during testing.

The approach we recommend is to set FOSWIKI_LIBS in your login script (e.g. .login, .csh, .profile depending on what shell you prefer).

EXPERTS: build.pl does not read bin/setlib.cfg. It uses $FOSWIKI_LIBS only to find the modules for the BuildContrib.

Build script

Each individual extension has its own build script, called build.pl, in its
module directory. A build script is a perl script that takes a number of
targets as its parameters. For example, perl build.pl test will run unit
tests, and perl build.pl release will build a new release.

The build script also accepts the following options:

-n

Do nothing; just print what you would have done

-N

Don't upload anything (used for testing target upload)

-v

Be verbose

-topiconly

with target upload, only upload the topic (not the archives)

Build targets are Perl functions, which operate on various data defined
in control files to build the various targets. Perl is used rather than
make for portability reasons.

print a guess at the MANIFEST (Caution - omits the primary System/Extension.txt topic

gitignore

print a guess at the .gitignore

=history=

Generates a list of svn checkins with comments suitable for use in the history section of the plugin/contrib topic. Not functional on a git repository.

dependencies

Find and print a best-guess dependencies list (for DEPENDENCIES)

The default target is test. The BuildContrib is designed so that most common behaviour is catered for. It is also easy to override any of the default targets in your build.pl and add extra behaviours.

The build target

Does nothing by default. This is the first target executed, and can be overridden by your build.pl to do something unusual - for example, executing an ANT file to build some Java.

The compress target

Usually only used if your extension includes Javascript or CSS, this target
minifies a file
to generate another that is functionally identical, but smaller for faster
download. It will search for an avaliable minifier:

To make the recommended compressor avaliable on your system, first install
Node.js, and then use "npm" to install the
compressors: npm install -g uglifyjs and npm install -g cssmin
This will install both uglify and MinCSS.

The compression can work with a number of different naming schemes, depending
on what files you refer to in your MANIFEST. For example, the name mappings
for javascript are:

XXX_src.js -> XXX.js

XXX_src.js -> XXX.compressed.js

XXX.uncompressed.js -> XXX.compressed.js

XXX.uncompressed.js -> XXX.js

XXX.js -> XXX.compressed.js

i.e. if you list XXX.compressed.js in MANIFEST, then the build will look
for XXX.uncompressed.js or XXX.js in the source tree to generate it from.
XXX.compressed.js will be regenerated even if it exists in the source tree
itself. If you list XXX.js in MANIFEST, then the build will look for a
XXX.uncompressed.js or XXX_src.js in the sources to generate it from.
The new files are generated in the source tree, so can be used for pseudo
installation and testing. However they are not checked in.

The easiest way to use compressed sources is to select the version your code is
to use based on a switch when you include the headers. For example, you can
use the DEBUG global from the Assert module:

When DEBUG is defined (i.e. when ASSERT is enabled), this will include
my.uncompressed.js and my.uncompressed.css, which makes debugging easier.
If DEBUG is not defined, it will include my.compressed.js and
my.compressed.css instead for best performance.

If you have your own debugging flag in your extension, you could use that
instead of DEBUG.

The tidy target

The test target

The test target is designed for use with extensions that have unit tests written using the UnitTestContrib. It automatically runs the unit tests found in the test/unit/extension_name directory.

The release target

The results of the release target are:

a Zip format archive,

a gzipped tar archive,

a md5 checksum,

the extension topic,

an installer script

The archives will each contain the following:

All the files listed in the MANIFEST

Another copy of the install/uninstall scripts

The tracked target

The tracked target is a special case of the release target. Given the
name of a customer, it will calculate a ciphered ID and replace
%$TRACKINGCODE% in the sources and documentation with the calculated
code. It will then make a release for that specific customer that
includes the tracking code. This is useful where you want
to be able to trace the code back to that customer without revealing any
details about them.

To use this target, you must manually add %$TRACKINGCODE% into your
sources and documentation.

The upload target

This target builds a release, and then tries to upload it to a target repository. The target uploads all the files in the release, and also tries to upload any attachments to the extension topic (as found by scanning the topic for META:FILEATTACHMENT).

You can control what server the upload is done to. This lets you - for example - set up your own corporate extensions server. Note that the upload operation will first attempt to download the topic to recover the PackageForm so that it can be added to the newly uploaded topic.
The upload also gives you a chance to specify an alternate download location
to be used for PackageForm recovery.

The manifest and dependencies targets

These are used when you are unsure of the correct contents of MANIFEST and DEPENDENCIES. They make a best-guess at the required contents for these files.

The twiki target

TWiki® is the precursor of Foswiki, and some support
for building extensions for TWiki is built in to the BuildContrib.

Firstly, extensions that were developed for use with the old TWiki
BuildContrib can be built using the Foswiki BuildContrib. Just modify
the build.pl to load the Foswiki build system rather than the old TWiki
build system.

Secondly, BuildContrib has a special target, twiki, which can be used
with a Foswiki build script to generate a TWiki directory structure and build
script, that can then be used to build an extension targeted at TWiki. The
files in the extension are run through a number of "mapping rules" that
will map much of the Foswiki namespace to TWiki. This transformation is not
complete, because Foswiki has many more features than TWiki, and because
CSS and Javascript cannot be reliably transformed this way. However many
extensions will work in TWiki after this transformation, and for others it
can be used as a launchpad for further manual mapping steps.

Example,

$ cd EditRowPlugin/lib/Foswiki/Plugins/EditRowPlugin
$ perl build.pl twiki
Created data/TWiki/EditRowPlugin.txt
Created lib/TWiki/Plugins/EditRowPlugin.pm
Created lib/TWiki/Plugins/EditRowPlugin/Table.pm
Created lib/TWiki/Plugins/EditRowPlugin/TableRow.pm
Created lib/TWiki/Plugins/EditRowPlugin/TableCell.pm
Created pub/TWiki/EditRowPlugin/screenshot.gif
Created pub/TWiki/EditRowPlugin/edittable.gif
Created pub/TWiki/EditRowPlugin/quiet.gif
Created pub/TWiki/EditRowPlugin/example3.gif
Created pub/TWiki/EditRowPlugin/example4.gif
Created pub/TWiki/EditRowPlugin/example5.gif
Created pub/TWiki/EditRowPlugin/addrow.gif
Created pub/TWiki/EditRowPlugin/TableSort.js
Created pub/TWiki/EditRowPlugin/TableSort_src.js
Created pub/TWiki/EditRowPlugin/erp.js
Created pub/TWiki/EditRowPlugin/erp_src.js
Created pub/TWiki/EditRowPlugin/wikiringlogo20x20.png
Created lib/TWiki/Plugins/EditRowPlugin/MANIFEST
Created lib/TWiki/Plugins/EditRowPlugin/DEPENDENCIES
Created lib/TWiki/Plugins/EditRowPlugin/build.pl
$ cd ../../../TWiki/Plugins/EditRowPlugin
$ perl build.pl release
Building a release for Version 0 (15 Feb 2009) of EditRowPlugin
MD5 checksums in EditRowPlugin/TWiki_EditRowPlugin.md5
.tgz in EditRowPlugin/TWiki_EditRowPlugin.tgz
.zip in EditRowPlugin/TWiki_EditRowPlugin.zip
WARNING: no .txt was generated
WARNING: no _installer was generated

There is no TWiki-specific topic generated. The Foswiki topic should suffice.
Installer generation is also disabled using !option installers none in
the tranformed MANIFEST. Users must install the generated TWiki packages
manually from the command-line. This is required due to bugs in TWiki.

Note the TWiki_ prefix on the archive names. This is useful to avoid naming
clashes with the standard Foswiki release of the same package. It is defined
using !option archive_prefix TWiki_ in the tranformed MANIFEST.

Extension authors are strongly recommended to check the functioning of the
TWiki versions of their extensions very carefully.

TWiki® is a trademark of Peter Thoeny.

MANIFEST

The MANIFEST file contains a list of all the files that are wanted in the
package. Each line is a file path, relative to the root of the installation.
Wildcards may NOT be used.
If the path contains spaces it must be enclosed in double-quotes.

Each file path has an optional octal permissions mask and a description.
For example,

If no permissions are given, permissions are guessed from the permissions on
the file in the source tree. These permissions are used by the installer
script to set file permissions in the installation.

The following permissions are recommended, and will be applied by default if
you don't specify anything different:

,v files. If you include a ,v file it will overwrite any existing ,v file when an extension is upgraded, potentially wiping out local changes on the end users installation. (The extension installer will not install ,v files even if they are included in the MANIFEST. However this is still a concern for anyone installing manually using unzip or tar. and sites using the new PlainFileStore will encounter errors if RCS ,v files are present.)

build.pl, MANIFEST, or any other side file used by the build process.

unit tests

Directives

MANIFEST files can contain a number of directives that are used to control
aspects of the build process. These directives always start with an
exclamation mark (!) and must be on a line on their own.

!ci and !noci

By default, files in the data and pub directories are
automatically checked in to Foswiki when the installation script is run
(for example, when installing from configure). This is useful when you
expect users to customise your files locally and you don't want to risk
overwriting their customisations. If you want to suppress this checkin
behaviour for individual files, you can add the string (noci) anywhere
in the description of the file. If you want to suppress it for larger
numbers of files, you can use the !noci and !ci directives in the
MANIFEST. Any files listed after a !noci directive, up to the next !ci
directive or the end of the file, will not be checked in when installing
to Foswiki 1.0.1 or later

!include

!include<path to extension directory>

MANIFESTs can also include other extensions that have been packaged using
BuildContrib. For example,

!include WysiwygPlugin/lib/Foswiki/Plugins/WysiwygPlugin

This will include the WysiwygPlugin in the release package.

Note that there is a script in the Foswiki tools directory called check_manifest.pl that can be run at any time to check the contents of your MANIFEST against what is checked into Subversion.

!include<path to extension MANIFEST* file >

When the target is a file, and not a directory, BuildContrib will recursively
process the MANIFEST file inline. This is useful when an extension packages a
complex 3rd party tool, for example, the TinyMCE Editor, and includes multiple
versions. Each copy of the tool can be included with it's own MANIFEST:

!option

!option is a general directive used to define global options. Currently
supported options are:

!option archive_prefix String_ will prefix the name of generated archive files with =String_

!option installer none will suppress the generation of an installer script.

DEPENDENCIES

The DEPENDENCIES file specifies dependencies on other extensions and
perl modules. Each line of the file is a single dependency:

name, version, type, description

where

name is the name of the module,

For Foswiki/TWiki/and CPAN modules, write the full module path, Foswiki::Plugins::MyPlugin

For external modules, use a "well known" name that can be understood by other package managers. Alternative packaging like .deb packages can resolve these external dependencies.

version is the version constraint (e.g. ">1.5"),

If version is written as r1234 (r followed by 1-6 digit number), the SVN release of the module will be compared, instead of the version.

Legacy dependencies - if written as a simple 4-digit number, will be assumed to be a SVN release number.

type is its type (cpan, perl, or eternal)

cpan modules should be found in the CPAN repositories.

perl modules include Foswiki:: and TWiki:: modules.

external or any other value is reported but ignored.

description is a short description of the module and where to get it. If it begins with the word Optional then the dependency will not be automatically resolved.

Dependencies of type cpan or perl will be executed in an eval statement to compare the VERSION and RELEASE strings.

The installer script written by the build process uses the dependency type to decide how to install dependant modules. 'cpan' means 'get the module from CPAN' and 'perl' means 'get the module from the Extensions web on Foswiki.org' (or whatever other repositories the admin has specified using $FOSWIKI_PACKAGES or $PLUGINS_URL).

Calculating DEPENDENCIES

When your module (the depender) depends on another module (a dependee), it is important to think carefully about what version of the dependee your module requires.

When you are working with Foswiki modules (such as contribs and plugins) you should list the version number of the module that you tested with. Normally you will want to use a > condition, so that more recent versions will also work. If a dependency on a Foswiki module fails (because the module isn't installed, for example) then the installer script will pull the latest version of the module from Foswiki.org, whether that is the required version or not. This is a limitation of the way plugins are stored on Foswiki.org.

As an alternative to using the Version number, you can also compare to the SVN release number. Write the version string as >=r1234. Note that the Version number is the preferred method, and is reported to the user during the install.

When you are working with CPAN modules, you need to take account of the fact that there are two types of CPAN modules; built-ins and add-ons.

Built-ins are perl modules that are pre-installed in the perl distribution. Since these modules are usually very stable, it is generally safe to express the version dependency as ">0" (i.e. "any version of the module will do").

Note however that the list of built-in modules is constantly growing with each new release of perl. So your module may be installed with a perl version that doesn't have the required module pre-installed. In this case, CPAN will automatically try to upgrade the perl version! There is no way around this, other than for the admin on the target system to manually install the module (download frm CPAN and build locally). You can help out the admin by expressing the dependency clearly, thus:

File::Find,>0,cpan,This module is shipped as part of standard perl from perl 5.8.0 onwards. If your perl installation is older than this, you should either upgrade perl, or manually install this module. If you allow this installer to continue, it will automatically upgrade your perl installation which is probably not what you want!

ONLYIF

A dependency may optionally be preceded by a condition that limits the cases
where the dependency applies. The condition is specified using a line that
contains ONLYIF ( condition ), where condition is a Perl
conditional. This is most useful for enabling dependencies only for certain
versions of other modules. For example,

File::Munge,>0,cpan,...
ONLYIF ( $Foswiki::Plugins::VERSION < 1.025)
MyPackage::FixOldFileFind, >=1.000, perl, Optional, only required if we have an old version of Foswiki API.

The ONLYIF only applies to the next dependency in the file.

Writing a build script

The easiest way to write a new build script is to use the
create_new_extension.pl script, which is part of the BuildContrib.

Create your plugin source tree

perl create_new_extension.pl BathPlugin

if you are working in a Subversion checkout, move the directory just created up one level to be at the same level as core.

Edit lib/Foswiki/Plugins/BathPlugin/MANIFEST and make sure it lists all the files you want to include in the release package

During development we recommend you use the pseudo-install.pl script to
soft-link your development code into your dev install. This script uses the
MANIFEST you write and creates softlinks (copies, on Windows) in your dev install that allow
you to run your test code without having to do a full re-install each time you
make a change.

If you have a pre-existing extension, and you want to package it for use with
BuildContrib, then you (may) need to create the module directory and write the
build.pl, MANIFEST and DEPENDENCIES files. The easiest way to do this
is to copy those files from an existing extension in git, and modify
them for your use.

Preparing the Installer

The installer script generated by the builder when target release is used is based on a template. This template is populated with lists of files and dependencies needed to make the extension-specific installer script.

PRE and POST processing scripts

You can extend this script by providing PREINSTALL, POSTINSTALL,
PREUNINSTALL, and/or POSTUNINSTALL files in the module directory.
These optional files are embedded into the template install script
at the appropriate stage of the installation. Read
lib/Foswiki/Contrib/BuildContrib/TEMPLATE_installer.pl (in the BuildContrib)
to see how they fit in.

With the Foswiki 1.1 version of the install tools, these exits run as methods
of the Configure::Package object instance for the extension, and have access
to the package manifest and other information. See the
PerlDoc for Foswiki::Configurer::Package for details (link requires BuildContrib to be installed).

If the script needs to report to the installer, it should return the message
as a simple string ending with a newline. It will be presented as a verbatim block to the web
install, or as inline text for the shell installation.

Caution: The pre/post scripts should not assume the standard installation
directories or topics when used to remove or otherwise manipulate files in the
installation. You can use the Utility function mapTarget to find the
correct file location for the current install. See
PerlDoc for Foswiki::Configure::Util (link requires BuildContrib to be installed)

Also, as the file to be removed is most likely not listed in the manifest,
it will not be backed up during the install. So use caution removing files
that would be required if fallback to the prior version of the plugin is
necessary.

For example, the POSTINSTALL script might look for an obsolete file from
a previous install, map it to the correct location for this installation, and
delete it if it exists.

my $this = shift; # Get the object instance passed to the routine
if ($this) { # Verify that you are running in the new environment
# Map the standard location to the absolute location on this
# installation of Foswiki. mapTarget is only available in Foswiki >= 1.1
my $mapped = Foswiki::Configure::Util::mapTarget( $this->{_rootdir}, 'tools/obsolete.pl');
my $count = unlink $mapped if ( -e $mapped ); # If it exists, delete it.
return "Removed $mapped \n " if ($count);
}

Testing

You are stongly recommended to develop a unit test suite for your extension.
Unit tests are kept in the test/unit/<name> directory for each extension.

To run the unit tests you will need to set up the test environment as described
in Foswiki::Development.GettingStarted

The easiest way to generate tests for your extension is to copy the approach taken in another extension. See for example ActionTrackerPlugin and CommentPlugin, which both have extensive test suites.

Tests are run using

perl build.pl test

Building a release

When you are almost ready to release, you should

Build a release package and installer

cd BathPlugin/lib/Foswiki/Plugins/BathPlugin

perl build.pl release

Remove the softlinked version from your dev install

cddev install

perl pseudo-install.pl -uninstall BathPlugin

Install the release package you just built:

cddev install

perl BathPlugin/BathPlugin_installer

Test.

Token expansion

The release target automatically expands certain tokens in .txt files
and in the installer script. The following tokens are supported:

Noteconfigure uses the | Version: | row in the table in the extension topic to determine what version of the package is installed. In the sources this is normally set to | Version: | 1.85 |. When you perl build.pl release, 1.85 is extracted from the primary Plugin or Contrib module, so it's very reliable and low maintenance. You don't have to use 1.85 in the | Version: | row of the extension topic - you can use your own version string if you want, or you can use 08 Jul 2015 which will take whatever value you have assigned to the $RELEASE variable in the extension master perl module. configure supports all of manually generated triples (1.2.3), ISO dates, and dd Mmmm yyy format dates as valid RELEASE identifiers.

Upload

When you are happy the release package is built correctly, you can
upload it.

cd BathPlugin/lib/Foswiki/Plugins/BathPlugin

perl build.pl upload

By default the upload target will upload to Foswiki.org. You will be prompted
to enter an alternate upload target, should you require it (e.g. to upload to
private corporate repository). The upload updates the topic and any associated
Var topics published by the extension, and uploads zip, tgz, md5 and installer
files.

Defaults for the upload, passwords, etc. are stored in the .buildcontrib
file in the users home directory.

Install support

Installer scripts build by BuildContrib are important for the full
functioning of the extensions installer in configure.

The installer script shipped with the package is very simple. By default all
it does is to check the dependencies you list, and if necessary download and
install any missing Foswiki and CPAN modules. Other dependencies are simply
checked. Topics shipped with the
module are automatically merged into any existing local copies, ensuring
histories are preserved.

If you want your installer to do anything else then you will need to write a POSTINSTALL script.

Installation Instructions

You are strongly recommended to use this Contrib to help split your code development away from your live Foswiki environment, as described above.

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. Use "Find More Extensions" to get a list of available extensions. Select "Install".