A Version Control System mantains a source tree, and tells the
buildmaster when it changes. The first step of each Build is typically
to acquire a copy of some version of this tree.

This chapter describes how the Buildbot learns about what Changes have
occurred. For more information on VC systems and Changes, see
Version Control Systems.

Changes can be provided by a variety of ChangeSource types, although any given
project will typically have only a single ChangeSource active. This section
provides a description of all available ChangeSource types and explains how to
set up each of them.

In general, each Buildmaster watches a single source tree. It is possible to
work around this, but true support for multi-tree builds remains elusive.

There are a variety of ChangeSource classes available, some of which are
meant to be used in conjunction with other tools to deliver Change
events from the VC repository to the buildmaster.

As a quick guide, here is a list of VC systems and the ChangeSources
that might be useful with them. Note that some of these modules are in
Buildbot's "contrib" directory, meaning that they have been offered by other
users in hopes they may be useful, and might require some additional work to
make them functional.

PBChangeSource (listening for connections from
monotone-buildbot.lua, which is available with monotone)

All VC systems can be driven by a PBChangeSource and the buildbotsendchange tool run from some form of commit script. If you write an email
parsing function, they can also all be driven by a suitable mail-parsing
source. Additionally, handlers for web-based
notification (i.e. from GitHub) can be used with WebStatus' change_hook module.
The interface is simple, so adding your own handlers (and sharing!) should be a
breeze.

ChangeSources will, in general, automatically provide the proper repository
attribute for any changes they produce. For systems which operate on URL-like
specifiers, this is a repository URL. Other ChangeSources adapt the concept as
necessary.

Many ChangeSources allow you to specify a project, as well. This attribute is
useful when building from several distinct codebases in the same buildmaster:
the project string can serve to differentiate the different codebases.
Schedulers can filter on project, so you can configure different builders to
run for each project.

Many projects publish information about changes to their source tree
by sending an email message out to a mailing list, frequently named
PROJECT-commits or PROJECT-changes. Each message usually contains a
description of the change (who made the change, which files were
affected) and sometimes a copy of the diff. Humans can subscribe to
this list to stay informed about what's happening to the source tree.

The Buildbot can also be subscribed to a -commits mailing list, and
can trigger builds in response to Changes that it hears about. The
buildmaster admin needs to arrange for these email messages to arrive
in a place where the buildmaster can find them, and configure the
buildmaster to parse the messages correctly. Once that is in place,
the email parser will create Change objects and deliver them to the
Schedulers (see Schedulers) just like any other ChangeSource.

There are two components to setting up an email-based ChangeSource.
The first is to route the email messages to the buildmaster, which is
done by dropping them into a maildir. The second is to actually
parse the messages, which is highly dependent upon the tool that was
used to create them. Each VC system has a collection of favorite
change-emailing tools, and each has a slightly different format, so
each has a different parsing function. There is a separate
ChangeSource variant for each parsing function.

Once you've chosen a maildir location and a parsing function, create
the change source and put it in change_source

The recommended way to install the buildbot is to create a dedicated
account for the buildmaster. If you do this, the account will probably
have a distinct email address (perhaps
buildmaster@example.org). Then just arrange for this
account's email to be delivered to a suitable maildir (described in
the next section).

If the buildbot does not have its own account, extension addresses
can be used to distinguish between email intended for the buildmaster
and email intended for the rest of the account. In most modern MTAs,
the e.g. foo@example.org account has control over every email
address at example.org which begins with "foo", such that email
addressed to account-foo@example.org can be delivered to a
different destination than account-bar@example.org. qmail
does this by using separate .qmail files for the two destinations
(.qmail-foo and .qmail-bar, with .qmail
controlling the base address and .qmail-default controlling all
other extensions). Other MTAs have similar mechanisms.

Thus you can assign an extension address like
foo-buildmaster@example.org to the buildmaster, and retain
foo@example.org for your own use.

A maildir is a simple directory structure originally developed for
qmail that allows safe atomic update without locking. Create a base
directory with three subdirectories: new, tmp, and cur.
When messages arrive, they are put into a uniquely-named file (using
pids, timestamps, and random numbers) in tmp. When the file is
complete, it is atomically renamed into new. Eventually the
buildmaster notices the file in new, reads and parses the
contents, then moves it into cur. A cronjob can be used to delete
files in cur at leisure.

Maildirs are frequently created with the maildirmake tool,
but a simple mkdir -p ~/MAILDIR/{cur,new,tmp} is pretty much
equivalent.

Many modern MTAs can deliver directly to maildirs. The usual .forward
or .procmailrc syntax is to name the base directory with a trailing
slash, so something like ~/MAILDIR/. qmail and postfix are
maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail
Delivery Agent).

If procmail is not setup on a system wide basis, then the following one-line
.forward file will invoke it.

!/usr/bin/procmail

For MTAs which cannot put files into maildirs directly, the
safecat tool can be executed from a .forward file to accomplish
the same thing.

The Buildmaster uses the linux DNotify facility to receive immediate
notification when the maildir's new directory has changed. When
this facility is not available, it polls the directory for new
messages, every 10 seconds by default.

The following sections describe the parsers available for each of
these tools.

Most of these parsers accept a prefix= argument, which is used
to limit the set of files that the buildmaster pays attention to. This
is most useful for systems like CVS and SVN which put multiple
projects in a single repository (or use repository names to indicate
branches). Each filename that appears in the email is tested against
the prefix: if the filename does not start with the prefix, the file
is ignored. If the filename does start with the prefix, that
prefix is stripped from the filename before any further processing is
done. Thus the prefix usually ends with a slash.

This parser works with the buildbot_cvs_maildir.py script in the
contrib directory.

The script sends an email containing all the files submitted in
one directory. It is invoked by using the CVSROOT/loginfo facility.

The Buildbot's CVSMaildirSource knows how to parse
these messages and turn them into Change objects. It takes two parameters,
the directory name of the maildir root, and an optional function to create
a URL for each file. The function takes three parameters:

file - file name
oldRev - old revision of the file
newRev - new revision of the file

For cvs version 1.12.x, the --path%p option is required.
Version 1.11.x and 1.12.x report the directory path differently.

The above example you put the buildbot_cvs_mail.py script under /cvsroot/CVSROOT.
It can be anywhere. Run the script with --help to see all the options.
At the very least, the
options -e (email) and -P (project) should be specified. The line must end with %{sVv}
This is expanded to the files that were modified.

Additional entries can be added to support more modules.

See buildbot_cvs_mail.py --help` for more information on the available options.

BzrLaunchpadEmailMaildirSource parses the mails that are sent to
addresses that subscribe to branch revision notifications for a bzr branch
hosted on Launchpad.

The branch name defaults to lp:Launchpadpath. For example
lp:~maria-captains/maria/5.1.

If only a single branch is used, the default branch name can be changed by
setting defaultBranch.

For multiple branches, pass a dictionary as the value of the branchMap
option to map specific repository paths to specific branch names (see example
below). The leading lp: prefix of the path is optional.

The prefix option is not supported (it is silently ignored). Use the
branchMap and defaultBranch instead to assign changes to
branches (and just do not subscribe the buildbot to branches that are not of
interest).

The revision number is obtained from the email text. The bzr revision id is
not available in the mails sent by Launchpad. However, it is possible to set
the bzr append_revisions_only option for public shared repositories to
avoid new pushes of merges changing the meaning of old revision numbers.

PBChangeSource actually listens on a TCP port for
clients to connect and push change notices into the
Buildmaster. This is used by the built-in buildbotsendchange
notification tool, as well as several version-control hook
scripts. This change is also useful for
creating new kinds of change sources that work on a push model
instead of some kind of subscription scheme, for example a script
which is run out of an email .forward file. This ChangeSource
always runs on the same TCP port as the slaves. It shares the same
protocol, and in fact shares the same space of "usernames", so you
cannot configure a PBChangeSource with the same name as a slave.

If you have a publicly accessible slave port, and are using
PBChangeSource, you must establish a secure username and password
for the change source. If your sendchange credentials are known (e.g., the
defaults), then your buildmaster is susceptible to injection of arbitrary
changes, which (depending on the build factories) could lead to arbitrary code
execution on buildslaves.

which port to listen on. If None (which is the default), it
shares the port used for buildslave connections.

user and passwd

The user/passwd account information that the client program must use
to connect. Defaults to change and changepw. Do not use
these defaults on a publicly exposed port!

prefix

The prefix to be found and stripped from filenames delivered over the
connection, defaulting to None. Any filenames which do not start with this prefix will be
removed. If all the filenames in a given Change are removed, the that
whole Change will be dropped. This string should probably end with a
directory separator.

This is useful for changes coming from version control systems that
represent branches as parent directories within the repository (like SVN
and Perforce). Use a prefix of trunk/ or
project/branches/foobranch/ to only follow one branch and to get
correct tree-relative filenames. Without a prefix, the
PBChangeSource will probably deliver Changes with filenames
like trunk/foo.c instead of just foo.c. Of course this also
depends upon the tool sending the Changes in (like buildbotsendchange) and what filenames it is delivering: that tool
may be filtering and stripping prefixes at the sending end.

Since Mercurial is written in python, the hook script can invoke
Buildbot's sendchange function directly, rather than having to
spawn an external process. This function delivers the same sort of
changes as buildbot sendchange and the various hook scripts in
contrib/, so you'll need to add a PBChangeSource to your
buildmaster to receive these changes.

To set this up, first choose a Mercurial repository that represents
your central official source tree. This will be the same
repository that your buildslaves will eventually pull from. Install
Buildbot on the machine that hosts this repository, using the same
version of python as Mercurial is using (so that the Mercurial hook
can import code from buildbot). Then add the following to the
.hg/hgrc file in that repository, replacing the buildmaster
hostname/portnumber as appropriate for your buildbot:

Mercurial lets you define multiple changegroup hooks by
giving them distinct names, like changegroup.foo and
changegroup.bar, which is why we use changegroup.buildbot
in this example. There is nothing magical about the buildbot
suffix in the hook name. The [hgbuildbot] section is special,
however, as it is the only section that the buildbot hook pays
attention to.)

Also note that this runs as a changegroup hook, rather than as
an incoming hook. The changegroup hook is run with
multiple revisions at a time (say, if multiple revisions are being
pushed to this repository in a single hg push command),
whereas the incoming hook is run with just one revision at a
time. The hgbuildbot.hook function will only work with the
changegroup hook.

If the buildmaster PBChangeSource is configured to require
sendchange credentials then you can set these with the auth
parameter. When this parameter is not set it defaults to
change:changepw, which are the defaults for the user and
password values of a PBChangeSource which doesn't require
authentication.

You can set this parameter in either the global /etc/mercurial/hgrc,
your personal ~/.hgrc file or the repository local .hg/hgrc
file. But since this value is stored in plain text, you must make sure that
it can only be read by those users that need to know the authentication
credentials.

The [hgbuildbot] section has two other parameters that you
might specify, both of which control the name of the branch that is
attached to the changes coming from this hook.

One common branch naming policy for Mercurial repositories is to use
it just like Darcs: each branch goes into a separate repository, and
all the branches for a single project share a common parent directory.
For example, you might have /var/repos/PROJECT/trunk/ and
/var/repos/PROJECT/release. To use this style, use the
branchtype=dirname setting, which simply uses the last
component of the repository's enclosing directory as the branch name:

[hgbuildbot]master=buildmaster.example.org:9987branchtype=dirname

Another approach is to use Mercurial's built-in branches (the kind
created with hg branch and listed with hg
branches). This feature associates persistent names with particular
lines of descent within a single repository. (note that the buildbot
source.Mercurial checkout step does not yet support this kind
of branch). To have the commit hook deliver this sort of branch name
with the Change object, use branchtype=inrepo:

[hgbuildbot]master=buildmaster.example.org:9987branchtype=inrepo

Finally, if you want to simply specify the branchname directly, for
all changes, use branch=BRANCHNAME. This overrides
branchtype:

[hgbuildbot]master=buildmaster.example.org:9987branch=trunk

If you use branch= like this, you'll need to put a separate
.hgrc in each repository. If you use branchtype=, you may be
able to use the same .hgrc for all your repositories, stored in
~/.hgrc or /etc/mercurial/hgrc.

As twisted needs to hook some Signals, and that some web server are
strictly forbiding that, the parameter fork in the
[hgbuildbot] section will instruct mercurial to fork before
sending the change request. Then as the created process will be of short
life, it is considered as safe to disable the signal restriction in
the Apache setting like that WSGIRestrictSignalOff. Refer to the
documentation of your web server for other way to do the same.

The category parameter sets the category for any changes generated from
the hook. Likewise, the project parameter sets the project. Changes'
repository attributes are formed from the Mercurial repo path by
stripping strip slashes.

Bzr is also written in Python, and the Bzr hook depends on Twisted to send the
changes.

To install, put contrib/bzr_buildbot.py in one of your plugins
locations a bzr plugins directory (e.g.,
~/.bazaar/plugins). Then, in one of your bazaar conf files (e.g.,
~/.bazaar/locations.conf), set the location you want to connect with buildbot
with these keys:

buildbot_on
one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via
commit, changes via push, or any changes to the trunk. 'change' is
recommended.

buildbot_server
(required to send to a buildbot master) the URL of the buildbot master to
which you will connect (as of this writing, the same server and port to which
slaves connect).

buildbot_port
(optional, defaults to 9989) the port of the buildbot master to which you will
connect (as of this writing, the same server and port to which slaves connect)

buildbot_pqm
(optional, defaults to not pqm) Normally, the user that commits the revision
is the user that is responsible for the change. When run in a pqm (Patch Queue
Manager, see https://launchpad.net/pqm) environment, the user that commits is
the Patch Queue Manager, and the user that committed the parent revision is
responsible for the change. To turn on the pqm mode, set this value to any of
(case-insensitive) "Yes", "Y", "True", or "T".

buildbot_dry_run
(optional, defaults to not a dry run) Normally, the post-commit hook will
attempt to communicate with the configured buildbot server and port. If this
parameter is included and any of (case-insensitive) "Yes", "Y", "True", or
"T", then the hook will simply print what it would have sent, but not attempt
to contact the buildbot master.

buildbot_send_branch_name
(optional, defaults to not sending the branch name) If your buildbot's bzr
source build step uses a repourl, do not turn this on. If your buildbot's
bzr build step uses a baseURL, then you may set this value to any of
(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master
append the branch name to the baseURL.

Note

The bzr smart server (as of version 2.2.2) doesn't know how
to resolve bzr:// urls into absolute paths so any paths in
locations.conf won't match, hence no change notifications
will be sent to Buildbot. Setting configuration parameters globally
or in-branch might still work. When buildbot no longer has a
hardcoded password, it will be a configuration option here as well.

Here's a simple example that you might have in your
~/.bazaar/locations.conf.

The P4Source periodically polls a Perforce
depot for changes. It accepts the following arguments:

p4base

The base depot path to watch, without the trailing '/...'.

p4port

The Perforce server to connect to (as host:port).

p4user

The Perforce user.

p4passwd

The Perforce password.

p4bin

An optional string parameter. Specify the location of the perforce command
line binary (p4). You only need to do this if the perforce binary is not
in the path of the buildbot user. Defaults to p4.

split_file

A function that maps a pathname, without the leading p4base, to a
(branch, filename) tuple. The default just returns (None,branchfile),
which effectively disables branch support. You should supply a function
which understands your repository structure.

pollinterval

How often to poll, in seconds. Defaults to 600 (10 minutes).

histmax

The maximum number of changes to inspect at a time. If more than this
number occur since the last poll, older changes will be silently
ignored.

encoding

The character encoding of p4's output. This defaults to "utf8", but
if your commit messages are in another encoding, specify that here.

This configuration uses the P4PORT, P4USER, and P4PASSWD
specified in the buildmaster's environment. It watches a project in which the
branch name is simply the next path component, and the file is all path
components after.

The BonsaiPoller periodically polls a Bonsai server. This is a
CGI script accessed through a web server that provides information
about a CVS tree, for example the Mozilla bonsai server at
http://bonsai.mozilla.org. Bonsai servers are usable by both
humans and machines. In this case, the buildbot's change source forms
a query which asks about any files in the specified branch which have
changed since the last query.

The base URL path to watch, like
svn://svn.twistedmatrix.com/svn/Twisted/trunk, or
http://divmod.org/svn/Divmo/, or even
file:///home/svn/Repository/ProjectA/branches/1.5/. This must
include the access scheme, the location of the repository (both the
hostname for remote ones, and any additional directory names necessary
to get to the repository), and the sub-path within the repository's
virtual filesystem for the project and branch of interest.

The SVNPoller will only pay attention to files inside the
subdirectory specified by the complete svnurl.

split_file

A function to convert pathnames into (branch,relative_pathname)
tuples. Use this to explain your repository's branch-naming policy to
SVNPoller. This function must accept a single string (the
pathname relative to the repository) and return a two-entry tuple. There
are a few utility functions in buildbot.changes.svnpoller that can
be used as a split_file function; see below for details.

The default value always returns (None,path), which indicates that
all files are on the trunk.

Subclasses of SVNPoller can override the split_file
method instead of using the split_file= argument.

project

Set the name of the project to be used for the SVNPoller. This
will then be set in any changes generated by the SVNPoller, and
can be used in a Change Filter for triggering
particular builders.

svnuser

An optional string parameter. If set, the --user argument will
be added to all svn commands. Use this if you have to
authenticate to the svn server before you can do svn info or
svn log commands.

svnpasswd

Like svnuser, this will cause a --password argument to
be passed to all svn commands.

pollinterval

How often to poll, in seconds. Defaults to 600 (checking once every 10
minutes). Lower this if you want the buildbot to notice changes
faster, raise it if you want to reduce the network and CPU load on
your svn server. Please be considerate of public SVN repositories by
using a large interval when polling them.

histmax

The maximum number of changes to inspect at a time. Every pollinterval
seconds, the SVNPoller asks for the last histmax changes and
looks through them for any revisions it does not already know about. If
more than histmax revisions have been committed since the last poll,
older changes will be silently ignored. Larger values of histmax will
cause more time and memory to be consumed on each poll attempt.
histmax defaults to 100.

svnbin

This controls the svn executable to use. If subversion is
installed in a weird place on your system (outside of the
buildmaster's PATH), use this to tell SVNPoller where
to find it. The default value of svn will almost always be
sufficient.

revlinktmpl

This parameter allows a link to be provided for each revision (for example,
to websvn or viewvc). These links appear anywhere changes are shown, such
as on build or change pages. The proper form for this parameter is an URL
with the portion that will substitute for a revision number replaced by
''%s''. For example, 'http://myserver/websvn/revision.php?rev=%s'
could be used to cause revision links to be created to a websvn repository
viewer.

cachepath

If specified, this is a pathname of a cache file that SVNPoller
will use to store its state between restarts of the master.

Several split file functions are available for common SVN repository layouts.
For a poller that is only monitoring trunk, the default split file function
is available explicitly as split_file_alwaystrunk:

If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To
use, put contrib/bzr_buildbot.py somewhere that your buildbot
configuration can import it. Even putting it in the same directory as the master.cfg
should work. Install the poller in the buildbot configuration as with any
other change source. Minimally, provide a URL that you want to poll (bzr://,
bzr+ssh://, or lp:), making sure the buildbot user has necessary
privileges.

# bzr_buildbot.py in the same directory as master.cfgfrombzr_buildbotimportBzrPollerc['change_source']=BzrPoller(url='bzr://hostname/my_project',poll_interval=300)

The BzrPoller parameters are:

url

The URL to poll.

poll_interval

The number of seconds to wait between polls. Defaults to 10 minutes.

branch_name

Any value to be used as the branch name. Defaults to None, or specify a
string, or specify the constants from bzr_buildbot.pySHORT or FULL to
get the short branch name or full branch address.

blame_merge_author

normally, the user that commits the revision is the user that is responsible
for the change. When run in a pqm (Patch Queue Manager, see
https://launchpad.net/pqm) environment, the user that commits is the Patch
Queue Manager, and the user that committed the merged, parent revision is
responsible for the change. set this value to True if this is pointed against
a PQM-managed branch.

If you cannot take advantage of post-receive hooks as provided by
contrib/git_buildbot.py for example, then you can use the GitPoller.

The GitPoller periodically fetches from a remote git repository and
processes any changes. It requires its own working directory for operation, which
can be specified via the workdir property. By default a temporary directory will
be used.

The GitPoller requires git-1.7 and later. It accepts the following
arguments:

repourl

the git-url that describes the remote repository, e.g.
git@example.com:foobaz/myrepo.git
(see the git fetch help for more info on git-url formats)

branch

the desired branch to fetch, will default to 'master'

workdir

the directory where the poller should keep its local repository. will
default to tempdir/gitpoller_work, which is probably not
what you want. If this is a relative path, it will be interpreted
relative to the master's basedir.

pollinterval

interval in seconds between polls, default is 10 minutes

gitbin

path to the git binary, defaults to just 'git'

fetch_refspec

One or more refspecs to use when fetching updates for the
repository. By default, the GitPoller will simply fetch
all refs. If your repository is large enough that this would be
unwise (or active enough on irrelevant branches that it'd be a
waste of time to fetch them all), you may wish to specify only a
certain refs to be updated. (A single refspec may be passed as a
string, or multiple refspecs may be passed as a list or set of
strings.)

category

Set the category to be used for the changes produced by the
GitPoller. This will then be set in any changes generated
by the GitPoller, and can be used in a Change Filter for
triggering particular builders.

project

Set the name of the project to be used for the
GitPoller. This will then be set in any changes generated
by the GitPoller, and can be used in a Change Filter for
triggering particular builders.

usetimestamps

parse each revision's commit timestamp (default is True),
or ignore it in favor of the current time (so recently processed
commits appear together in the waterfall page)

encoding

Set encoding will be used to parse author's name and commit
message. Default encoding is 'utf-8'. This will not be
applied to file names since git will translate non-ascii file
names to unreadable escape sequences.

This class adds a change to the buildbot system for each of the following events:

patchset-created

A change is proposed for review. Automatic checks like
checkpatch.pl can be automatically triggered. Beware of
what kind of automatic task you trigger. At this point, no trusted
human has reviewed the code, and a patch could be specially
crafted by an attacker to compromise your buildslaves.

ref-updated

A change has been merged into the repository. Typically, this kind
of event can lead to a complete rebuild of the project, and upload
binaries to an incremental build results server.

This class will populate the property list of the triggered build with the info
received from Gerrit server in JSON format.

The GoogleCodeAtomPoller periodically polls a Google Code Project's
commit feed for changes. Works on SVN, Git, and Mercurial repositories. Branches
are not understood (yet). It accepts the following arguments: