Project description

Buildout is a project designed to solve 2 problems:

Application-centric assembly and deployment

Assembly runs the gamut from stitching together libraries to
create a running program, to production deployment configuration of
applications, and associated systems and tools (e.g. run-control
scripts, cron jobs, logs, service registration, etc.).

Buildout might be confused with build tools like make or ant, but
it is a little higher level and might invoke systems like make or
ant to get it’s work done.

Buildout might be confused with systems like puppet or chef, but it
is more application focused. Systems like puppet or chef might
use buildout to get their work done.

Buildout is also somewhat Python-centric, even though it can be
used to assemble and deploy non-python applications. It has some
special features for assembling Python programs. It’s scripted with
Python, unlike, say puppet or chef, which are scripted with Ruby.

Repeatable assembly of programs from Python software distributions

Buildout puts great effort toward making program assembly a highly
repeatable process, whether in a very open-ended development mode,
where dependency versions aren’t locked down, or in a deployment
environment where dependency versions are fully specified. You
should be able to check buildout into a VCS and later check it out.
Two checkouts built at the same time in the same environment should
always give the same result, regardless of their history. Among
other things, after a buildout, all dependencies should be at the
most recent version consistent with any version specifications
expressed in the buildout.

Buildout supports applications consisting of multiple programs,
with different programs in an application free to use different
versions of Python distributions. This is in contrast with a
Python installation (real or virtual), where, for any given
distribution, there can only be one installed.

There are a number of ways to install buildout. You can install it as
you would any other package, using pip or easy_install. In this case,
you’ll get a buildout command that you can use to build projects. To
build a project, just use:

buildout

from a project directory.

Buildout’s (stubborn) philosophy, however, is that projects should be
self-contained, and not require changes to a shared Python
installation. To avoid changing a shared Python installation you can
download a bootstrap script that, when run, will install buildout
locally in your project.

buildout 2 is somewhat backward-incompatible with version 1. Most
projects will probably work fine with either. If you need to keep
using version 1, however, specify a version requirement when you use
pip or easy_install, or use the version 1 bootstrap script at:

Below, you’ll find doctest-based documentation. It was an experiment
in reusing tests as documentation. The experiment didn’t go that
well, but there may be details below that aren’t easy to find on
buildout.org yet.

The word “buildout” refers to a description of a set of parts and the
software to create and assemble them. It is often used informally to
refer to an installed system based on a buildout definition. For
example, if we are creating an application named “Foo”, then “the Foo
buildout” is the collection of configuration and application-specific
software that allows an instance of the application to be created. We
may refer to such an instance of the application informally as “a Foo
buildout”.

This document describes how to define buildouts using buildout
configuration files and recipes. There are three ways to set up the
buildout software and create a buildout instance:

Install the zc.buildout egg with easy_install and use the buildout
script installed in a Python scripts area.

Use the buildout bootstrap script to create a buildout that
includes both the setuptools and zc.buildout eggs. This allows you
to use the buildout software without modifying a Python install.
The buildout script is installed into your buildout local scripts
area.

Use a buildout command from an already installed buildout to
bootstrap a new buildout. (See the section on bootstraping later
in this document.)

Often, a software project will be managed in a software repository,
such as a subversion repository, that includes some software source
directories, buildout configuration files, and a copy of the buildout
bootstrap script. To work on the project, one would check out the
project from the repository and run the bootstrap script which
installs setuptools and zc.buildout into the checkout as well as any
parts defined.

We have a sample buildout that we created using the bootstrap command
of an existing buildout (method 3 above). It has the absolute minimum
information. We have bin, develop-eggs, eggs and parts directories,
and a configuration file:

The develop-eggs directory holds egg links for software being
developed in the buildout. We separate develop-eggs and other eggs to
allow eggs directories to be shared across multiple buildouts. For
example, a common developer technique is to define a common eggs
directory in their home that all non-develop eggs are stored in. This
allows larger buildouts to be set up much more quickly and saves disk
space.

The parts directory provides an area where recipes can install
part data. For example, if we built a custom Python, we would
install it in the part directory. Part data is stored in a
sub-directory of the parts directory with the same name as the part.

Buildouts are defined using configuration files. These are in the
format defined by the Python ConfigParser module, with extensions
that we’ll describe later. By default, when a buildout is run, it
looks for the file buildout.cfg in the directory where the buildout is
run.

The minimal configuration file has a buildout section that defines no
parts:

>>> cat(sample_buildout, 'buildout.cfg')
[buildout]
parts =

A part is simply something to be created by a buildout. It can be
almost anything, such as a Python package, a program, a directory, or
even a configuration file.

A part is created by a recipe. Recipes are always installed as Python
eggs. They can be downloaded from a package server, such as the
Python Package Index, or they can be developed as part of a project
using a “develop” egg.

A develop egg is a special kind of egg that gets installed as an “egg
link” that contains the name of a source directory. Develop eggs
don’t have to be packaged for distribution to be used and can be
modified in place, which is especially useful while they are being
developed.

Let’s create a recipe as part of the sample project. We’ll create a
recipe for creating directories. First, we’ll create a recipes source
directory for our local recipes:

The constructor is responsible for updating a parts options to reflect
data read from other sections. The buildout system keeps track of
whether a part specification has changed. A part specification has
changed if it’s options, after adjusting for data read from other
sections, has changed, or if the recipe has changed. Only the options
for the part are considered. If data are read from other sections,
then that information has to be reflected in the parts options. In
the Mkdir example, the given path is interpreted relative to the
buildout directory, and data from the buildout directory is read. The
path option is updated to reflect this. If the directory option was
changed in the buildout sections, we would know to update parts
created using the mkdir recipe using relative path names.

When buildout is run, it saves configuration data for installed parts
in a file named “.installed.cfg”. In subsequent runs, it compares
part-configuration data stored in the .installed.cfg file and the
part-configuration data loaded from the configuration files as
modified by recipe constructors to decide if the configuration of a
part has changed. If the configuration has changed, or if the recipe
has changed, then the part is uninstalled and reinstalled. The
buildout only looks at the part’s options, so any data used to
configure the part needs to be reflected in the part’s options. It is
the job of a recipe constructor to make sure that the options include
all relevant data.

Of course, parts are also uninstalled if they are no-longer used.

The recipe defines a constructor that takes a buildout object, a part
name, and an options dictionary. It saves them in instance attributes.
If the path is relative, we’ll interpret it as relative to the
buildout directory. The buildout object passed in is a mapping from
section name to a mapping of options for that section. The buildout
directory is available as the directory option of the buildout
section. We normalize the path and save it back into the options
directory.

The install method is responsible for creating the part. In this
case, we need the path of the directory to create. We’ll use a path
option from our options dictionary. The install method logs what it’s
doing using the Python logging call. We return the path that we
installed. If the part is uninstalled or reinstalled, then the path
returned will be removed by the buildout machinery. A recipe install
method is expected to return a string, or an iterable of strings
containing paths to be removed if a part is uninstalled. For most
recipes, this is all of the uninstall support needed. For more complex
uninstallation scenarios use Uninstall recipes.

The update method is responsible for updating an already installed
part. An empty method is often provided, as in this example, if parts
can’t be updated. An update method can return None, a string, or an
iterable of strings. If a string or iterable of strings is returned,
then the saved list of paths to be uninstalled is updated with the new
information by adding any new files returned by the update method.

We need to provide packaging information so that our recipe can be
installed as a develop egg. The minimum information we need to specify
is a name. For recipes, we also need to define the
names of the recipe classes as entry points. Packaging information is
provided via a setup.py script:

Our setup script defines an entry point. Entry points provide
a way for an egg to define the services it provides. Here we’ve said
that we define a zc.buildout entry point named mkdir. Recipe
classes must be exposed as entry points in the zc.buildout group. we
give entry points names within the group.

We also need a README.txt for our recipes to avoid an annoying warning
from distutils, on which setuptools and zc.buildout are based:

This tells the buildout to install a development egg for our recipes.
Any number of paths can be listed. The paths can be relative or
absolute. If relative, they are treated as relative to the buildout
directory. They can be directory or file paths. If a file path is
given, it should point to a Python setup script. If a directory path
is given, it should point to a directory containing a setup.py file.
Development eggs are installed before building any parts, as they may
provide locally-defined recipes needed by the parts.

parts = data-dir

Here we’ve named a part to be “built”. We can use any name we want
except that different part names must be unique and recipes will often
use the part name to decide what to do.

[data-dir]
recipe = recipes:mkdir
path = mystuff

When we name a part, we also create a section of the same
name that contains part data. In this section, we’ll define
the recipe to be used to install the part. In this case, we also
specify the path to be created.

Let’s run the buildout. We do so by running the build script in the
buildout:

If a user makes an error, an error needs to be printed and work needs
to stop. This is accomplished by logging a detailed error message and
then raising a (or an instance of a subclass of a)
zc.buildout.UserError exception. Raising an error other than a
UserError still displays the error, but labels it as a bug in the
buildout software or recipe. In the sample above, of someone gives a
non-existent directory to create the directory in:

It’s critical that recipes clean up partial effects when errors
occur. Because recipes most commonly create files and directories,
buildout provides a helper API for removing created files when an
error occurs. Option objects have a created method that can be called
to record files as they are created. If the install or update method
returns with an error, then any registered paths are removed
automatically. The method returns the files registered and can be
used to return the files created. Let’s use this API to simplify the
recipe:

A buildout configuration file consists of a sequence of sections. A
section has a section header followed by 0 or more section options.
(Buildout configuration files may be viewed as a variation on INI
files.)

A section header consists of a section name enclosed in square braces.
A section name consists of one or more non-whitespace characters other
than square braces (‘[‘, ‘]’), curly braces (‘{‘, ‘}’), colons (‘:’)
or equal signs (‘=’). Whitespace surrounding section names is ignored.

A section header can optionally have a condition expression separated
by a colon. See Conditional sections.

Options consist of option names, followed by optional space or tab
characters, an optional plus or minus sign and an equal signs and
values. An option value may be spread over multiple lines as long as
the lines after the first start with a whitespace character. An
option name consists of one or more non-whitespace characters other
than equal signs, square braces (“[“, “]”), curly braces (“{“, “}”),
plus signs or colons (“:”). The option name ‘<’ is reserved. An
option’s data consists of the characters following the equal sign on
the start line, plus the continuation lines.

Option values have extra whitespace stripped. How this is done
depends on whether the value has non-whitespace characterts on the
first line. If an option value has non-whitespace characters on the
first line, then each line is stripped and blank lines are removed.
For example, in:

[foo]
bar = 1
baz = a
b
c

The value of of bar is '1' and the value of baz is
'a\nb\nc'.

If the first line of an option doesn’t contain whitespace, then the
value is dedented (with textwrap.dedent), trailing spaces in lines
are removed, and leading and trailing blank lines are removed. For
example, in:

[foo]
bar =
baz =
a
b
c

The value of bar is '', and the value of baz is 'a\n b\n\nc'.

Lines starting with ‘#’ or ‘;’ characters are comments. Comments can
also be placed after the closing square bracket (‘]’) in a section header.

Buildout configuration data are Python strings, which are bytes in
Python 2 and unicode in Python 3.

Sections and options within sections may be repeated. Multiple
occurrences of of a section are treated as if they were concatenated.
The last option value for a given name in a section overrides previous
values.

In addition top the syntactic details above:

option names are case sensitive

option values can use a substitution syntax, described below, to
refer to option values in specific sections.

When used with the annotate command, buildout displays annotated sections.
All sections are displayed, sorted alphabetically. For each section,
all key-value pairs are displayed, sorted alphabetically, along with
the origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE,
COMMAND_LINE_VALUE).

We used a string-template substitution for File-1 and File-2. This
type of substitution uses the string.Template syntax. Names
substituted are qualified option names, consisting of a section name
and option name joined by a colon.

Now, if we run the buildout, we’ll see the options with the values
substituted.

Note that the substitution of the data-dir path option reflects the
update to the option performed by the mkdir recipe.

It might seem surprising that mydata was created again. This is
because we changed our recipes package by adding the debug module.
The buildout system didn’t know if this module could effect the mkdir
recipe, so it assumed it could and reinstalled mydata. If we rerun
the buildout:

When a section with a recipe is referred to, either through variable
substitution or by an initializing recipe, the section is treated as a
part and added to the part list before the referencing part. For
example, we can leave data-dir out of the parts list:

A section (other than the buildout section) can extend one or more
other sections using the < option. Options from the referenced
sections are copied to the referring section before variable
substitution. This, together with the ability to refer to variables
of the current section allows sections to be used as macros.

Sometimes, you need different configuration in different environments
(different operating systems, or different versions of Python). To
make this easier, you can define environment-specific options by
providing conditional sections:

[ctl]
suffix =
[ctl:windows]
suffix = .bat

In this tiny example, we’ve defined a ctl:suffix option that’s
.bat on Windows and an empty string elsewhere.

A conditional section has a colon and then a Python expression after
the name. If the Python expression result is true, the section
options from the section are included. If the value is false, the
section is ignored.

Some things to note:

If there is no exception, then options from the section are
included.

Sections and options can be repeated. If an option is repeated, the
last value is used. In the example above, on Windows, the second
suffix option overrides the first. If the order of the sections
was reversed, the conditional section would have no effect.

In addition to the normal built-ins, the expression has access to
global variable that make common cases short and description as shown
above:

The example is pretty trivial, but the pattern it illustrates is
pretty common. In a more practical example, the base buildout might
represent a product and the extending buildout might be a
customization.

Here we specified a URL for the file we extended. The file we
downloaded, itself referred to a file on the server using a relative
URL reference. Relative references are interpreted relative to the
base URL when they appear in configuration files loaded via URL.

We can also specify a URL as the configuration file to be used by a
buildout.

Normally, the buildout directory defaults to directory
containing a configuration file. This won’t work for configuration
files loaded from URLs. In this case, the buildout directory would
normally be defined on the command line:

If the file $HOME/.buildout/default.cfg, exists, it is read before
reading the configuration file. ($HOME is the value of the HOME
environment variable. The ‘/’ is replaced by the operating system file
delimiter.)

As we’ve seen, when parts are installed, buildout keeps track of files
and directories that they create. When the parts are uninstalled these
files and directories are deleted.

Sometimes more clean up is needed. For example, a recipe might add a
system service by calling chkconfig –add during installation. Later
during uninstallation, chkconfig –del will need to be called to
remove the system service.

In order to deal with these uninstallation issues, you can register
uninstall recipes. Uninstall recipes are registered using the
‘zc.buildout.uninstall’ entry point. Parts specify uninstall recipes
using the ‘uninstall’ option.

In comparison to regular recipes, uninstall recipes are much
simpler. They are simply callable objects that accept the name of the
part to be uninstalled and the part’s options dictionary. Uninstall
recipes don’t have access to the part itself since it maybe not be
able to be instantiated at uninstallation time.

Here’s a recipe that simulates installation of a system service, along
with an uninstall recipe that simulates removing the service.

To use these recipes we must register them using entry points. Make
sure to use the same name for the recipe and uninstall recipe. This is
required to let buildout know which uninstall recipe goes with which
recipe.

Uninstall recipes don’t have to take care of removing all the files
and directories created by the part. This is still done automatically,
following the execution of the uninstall recipe. An upshot is that an
uninstallation recipe can access files and directories created by a
recipe before they are deleted.

For example, here’s an uninstallation recipe that simulates backing up
a directory before it is deleted. It is designed to work with the
mkdir recipe introduced earlier.

A number of arguments can be given on the buildout command line. The
command usage is:

buildout [options and assignments] [command [command arguments]]

The following options are supported:

-h (or –help)

Print basic usage information. If this option is used, then all
other options are ignored.

-c filename

The -c option can be used to specify a configuration file, rather than
buildout.cfg in the current directory.

-t socket_timeout

Specify the socket timeout in seconds.

-v

Increment the verbosity by 10. The verbosity is used to adjust
the logging level. The verbosity is subtracted from the numeric
value of the log-level option specified in the configuration file.

-q

Decrement the verbosity by 10.

-U

Don’t read user-default configuration.

-o

Run in off-line mode. This is equivalent to the assignment
buildout:offline=true.

-O

Run in non-off-line mode. This is equivalent to the assignment
buildout:offline=false. This is the default buildout mode. The
-O option would normally be used to override a true offline
setting in a configuration file.

-n

Run in newest mode. This is equivalent to the assignment
buildout:newest=true. With this setting, which is the default,
buildout will try to find the newest versions of distributions
available that satisfy its requirements.

-N

Run in non-newest mode. This is equivalent to the assignment
buildout:newest=false. With this setting, buildout will not seek
new distributions if installed distributions satisfy it’s
requirements.

The most commonly used command is ‘install’ and it takes a list of
parts to install. if any parts are specified, only those parts are
installed. To illustrate this, we’ll update our configuration and run
the buildout in the usual way:

The buildout normally puts the bin, eggs, and parts directories in the
directory in the directory containing the configuration file. You can
provide alternate locations, and even names for these directories.

Here, we’ve changed the format to include the log-level name, rather
than the logger name.

We’ve also illustrated, with a contrived example, that the log level
can be a numeric value and that the verbosity can be specified in the
configuration file. Because the verbosity is subtracted from the log
level, we get a final log level of 20, which is the INFO level.

Buildouts have a number of predefined options that recipes can use
and that users can override in their configuration files. To see
these, we’ll run a minimal buildout configuration with a debug logging
level. One of the features of debug logging is that the configuration
database is shown.

All of these options can be overridden by configuration files or by
command-line assignments. We’ve discussed most of these options
already, but let’s review them and touch on some we haven’t discussed:

allow-hosts

On some environments the links visited by zc.buildout can be forbidden by
paranoid firewalls. These URLs might be in the chain of links visited by
zc.buildout as defined by buildout’s find-links option, or as defined
by various eggs in their url, download_url, dependency_links metadata.

The fact that package_index works like a spider and might visit links and
go to other locations makes this even harder.

The allow-hosts option provides a way to prevent this, and
works exactly like the one provided in easy_install.

You can provide a list of allowed host, together with wildcards:

[buildout]
...
allow-hosts =
*.python.org
example.com

All URLs that does not match these hosts will not be visited.

allow-picked-versions

By default, the buildout will choose the best match for a given requirement
if the requirement is not specified precisely (for instance, using the
“versions” option. This behavior corresponds to the
“allow-picked-versions” being set to its default value, “true”. If
“allow-picked-versions” is “false,” instead of picking the best match,
buildout will raise an error. This helps enforce repeatability.

bin-directory

The directory path where scripts are written. This can be a
relative path, which is interpreted relative to the directory
option.

develop-eggs-directory

The directory path where development egg links are created for software
being created in the local project. This can be a relative path,
which is interpreted relative to the directory option.

directory

The buildout directory. This is the base for other buildout file
and directory locations, when relative locations are used.

eggs-directory

The directory path where downloaded eggs are put. It is common to share
this directory across buildouts. Eggs in this directory should
never be modified. This can be a relative path, which is
interpreted relative to the directory option.

find-links

You can specify more locations to search for distributions using the
find-links option. All locations specified will be searched for
distributions along with the package index as described before.

A download cache can be used as the basis of application source releases.
In an application source release, we want to distribute an application that
can be built without making any network accesses. In this case, we
distribute a buildout with download cache and tell the buildout to install
from the download cache only, without making network accesses. The
buildout install-from-cache option can be used to signal that packages
should be installed only from the download cache.

installed

The file path where information about the results of the previous
buildout run is written. This can be a relative path, which is
interpreted relative to the directory option. This file provides
an inventory of installed parts with information needed to decide
which if any parts need to be uninstalled.

log-format

The format used for logging messages.

log-level

The log level before verbosity adjustment

newest

By default buildout and recipes will try to find the newest versions of
distributions needed to satisfy requirements. This can be very time
consuming, especially when incrementally working on setting up a buildout
or working on a recipe. The buildout “newest” option can be used to to
suppress this. If the “newest” option is set to false, then new
distributions won’t be sought if an installed distribution meets
requirements. The “newest” option can also be set to false using the -N
command-line option. See also the “offline” option.

offline

The “offline” option goes a bit further than the “newest” option. If the
buildout “offline” option is given a value of “true”, the buildout and
recipes that are aware of the option will avoid doing network access. This
is handy when running the buildout when not connected to the internet. It
also makes buildouts run much faster. This option is typically set using
the buildout -o option.

parts

A white space separated list of parts to be installed.

parts-directory

A working directory that parts can used to store data.

prefer-final

Currently, when searching for new releases, the newest available
release is used. This isn’t usually ideal, as you may get a
development release or alpha releases not ready to be widely used.
You can request that final releases be preferred using the prefer
final option in the buildout section:

[buildout]
...
prefer-final = true

When the prefer-final option is set to true, then when searching for
new releases, final releases are preferred. If there are final
releases that satisfy distribution requirements, then those releases
are used even if newer non-final releases are available. The buildout
prefer-final option can be used to override this behavior.

In buildout version 2, final releases will be preferred by default.
You will then need to use a false value for prefer-final to get the
newest releases.

use-dependency-links

By default buildout will obey the setuptools dependency_links metadata
when it looks for dependencies. This behavior can be controlled with
the use-dependency-links buildout option:

[buildout]
...
use-dependency-links = false

The option defaults to true. If you set it to false, then dependency
links are only looked for in the locations specified by find-links.

verbosity

A log-level adjustment. Typically, this is set via the -q and -v
command-line options.

(We list both the eggs and develop-eggs directories because the
buildout or setuptools egg could be installed in the develop-eggs
directory if the original buildout had develop eggs for either
buildout or setuptools.)

Note that the buildout script was installed but not run. To run
the buildout, we’d have to run the installed buildout script.

If we have an existing buildout that already has a buildout.cfg, we’ll
normally use the bootstrap command instead of init. It will complain
if there isn’t a configuration file:

Passing requirements or paths causes the the buildout to be run as part
of initialization. In the example above, we got a number of
distributions installed and 2 scripts generated. The first, demo,
was defined by the demo project. The second, py was defined by
the generated configuration. It’s a “custom interpreter” that behaves
like a standard Python interpreter, except that includes the specified
eggs and extra paths in it’s Python path.

We specified a source directory that didn’t exist. Buildout created it
for us:

By default, buildout searches the Python Package Index when looking
for distributions. You can, instead, specify your own index to search
using the index option:

[buildout]
...
index = http://index.example.com/

This index, or the default of http://pypi.python.org/simple/ if no
index is specified, will always be searched for distributions unless
running buildout with options that prevent searching for
distributions. The latest version of the distribution that meets the
requirements of the buildout will always be used.

You can also specify more locations to search for distributions using
the find-links option. See its description above.

The buildout installed option is used to specify the file used to save
information on installed parts. This option is initialized to
“.installed.cfg”, but it can be overridden in the configuration file
or on the command line:

A feature allows code to be loaded and run after
configuration files have been read but before the buildout has begun
any processing. The intent is to allow special plugins such as
urllib2 request handlers to be loaded.

To load an extension, we use the extensions option and list one or
more distribution requirements, on separate lines. The distributions
named will be loaded and any zc.buildout.extension entry points found
will be called with the buildout as an argument. When buildout
finishes processing, any zc.buildout.unloadextension entry points
found will be called with the buildout as an argument.

Let’s create a sample extension in our sample buildout created in the
previous section:

Now we can add the extensions option. We were a bit tricky and ran
the buildout once with the demo develop egg defined but without the
extension option. This is because extensions are loaded before the
buildout creates develop eggs. We needed to use a separate buildout
run to create the develop egg. Normally, when eggs are loaded from
the network, we wouldn’t need to do anything special.

One of the goals of zc.buildout is to provide enough control to make
buildouts repeatable. It should be possible to check the buildout
configuration files for a project into a version control system and
later use the checked in files to get the same buildout, subject to
changes in the environment outside the buildout.

An advantage of using Python eggs is that dependencies of eggs used are
automatically determined and used. The automatic inclusion of
dependent distributions is at odds with the goal of repeatable
buildouts.

To support repeatable buildouts, a versions section can be created
with options for each distribution name who’s version is to be fixed.
The section can then be specified via the buildout versions option.

The zc.buildout.download module provides a download utility that handles
the details of downloading files needed for a buildout run from the internet.
It downloads files to the local file system, using the download cache if
desired and optionally checking the downloaded files’ MD5 checksum.

Downloading a file is achieved by calling the utility with the URL as an
argument. A tuple is returned that consists of the path to the downloaded copy
of the file and a boolean value indicating whether this is a temporary file
meant to be cleaned up during the same buildout run:

It is common to store cached copies of downloaded files within sub-directories
of the download cache to keep some degree of order. For example, zc.buildout
stores downloaded distributions in a sub-directory named “dist”. Those
sub-directories are also known as namespaces. So far, we haven’t specified any
namespaces to use, so the download utility stored files directly inside the
download cache. Let’s use a namespace “test” instead:

The next time we want to download that file, the copy from inside the cache
namespace is used. To see this clearly, we put a file with the same name but
different content both on the server and in the cache’s root directory:

So far, the base name of the downloaded file read from the URL has been used
for the name of the cached copy of the file. This may not be desirable in some
cases, for example when downloading files from different locations that have
the same base name due to some naming convention, or if the file content
depends on URL parameters. In such cases, an MD5 hash of the complete URL may
be used as the filename in the cache:

The path was printed just to illustrate matters; we cannot know the real
checksum since we don’t know which port the server happens to listen at when
the test is run, so we don’t actually know the full URL of the file. Let’s
check that the checksum actually belongs to the particular URL used:

Sometimes it is desirable to try downloading a file from the net if at all
possible, and use the cache purely as a fall-back option when a server is
down or if we are in offline mode. This mode is only in effect if a download
cache is configured in the first place:

The configuration options explained so far derive from the build logic
implemented by the calling code. Other options configure the download utility
for use in a particular project or buildout run; they are read from the
buildout configuration section. The latter can be passed directly as the
first argument to the download utility’s constructor.

The location of the download cache is specified by the download-cache
option:

When “downloading” a directory given by file-system path or file: URL and
using a download cache at the same time, the cached directory wasn’t handled
correctly. Consequently, the cache was defeated and an attempt to cache the
directory a second time broke. This is how it should work:

Normally, when distributions are installed, if any processing is
needed, they are downloaded from the internet to a temporary directory
and then installed from there. A download cache can be used to avoid
the download step. This can be useful to reduce network access and to
create source distributions of an entire buildout.

The buildout download-cache option can be used to specify a directory
to be used as a download cache.

A download cache can be used as the basis of application source
releases. In an application source release, we want to distribute an
application that can be built without making any network accesses. In
this case, we distribute a buildout with download cache and tell the
buildout to install from the download cache only, without making
network accesses. The buildout install-from-cache option can be used
to signal that packages should be installed only from the download
cache.

Let’s remove our installed eggs and run the buildout with the
install-from-cache option set to true:

As mentioned in the general buildout documentation, configuration files can
extend each other, including the ability to download configuration being
extended from a URL. If desired, zc.buildout caches downloaded configuration
in order to be able to use it when run offline.

As we’re going to talk about downloading things, let’s start an HTTP server.
Also, all of the following will take place inside the sample buildout.

Let’s now specify a cache for base configuration files. This cache is
different from the download cache used by recipes for caching distributions
and other files; one might, however, use a namespace subdirectory of the
download cache for it. The configuration cache we specify will be created when
running buildout and the base.cfg file will be put in it (with the file name
being a hash of the complete URL):

The cache is being used purely as a fall-back in case we are offline or don’t
have access to a configuration file to be downloaded. As long as we are
online, buildout attempts to download a fresh copy of each file even if a
cached copy of the file exists. To see this, we put different configuration in
the same place on the server and run buildout in offline mode so it takes
base.cfg from the cache:

Normally, the values of buildout options such as the location of a download
cache or whether to use offline mode are determined by first reading the
user’s default configuration, updating it with the project’s configuration and
finally applying command-line options. User and project configuration are
assembled by reading a file such as ~/.buildout/default.cfg,
buildout.cfg or a URL given on the command line, recursively (depth-first)
downloading any base configuration specified by the buildout:extends
option read from each of those config files, and finally evaluating each
config file to provide default values for options not yet read.

This works fine for all options that do not influence how configuration is
downloaded in the first place. The extends-cache and offline options,
however, are treated differently from the procedure described in order to make
it simple and obvious to see where a particular configuration file came from
under any particular circumstances.

Offline and extends-cache settings are read from the two root config files
exclusively. Otherwise one could construct configuration files that, when
read, imply that they should have been read from a different source than
they have. Also, specifying the extends cache within a file that might have
to be taken from the cache before being read wouldn’t make a lot of sense.

Offline and extends-cache settings given by the user’s defaults apply to the
process of assembling the project’s configuration. If no extends cache has
been specified by the user’s default configuration, the project’s root
config file must be available, be it from disk or from the net.

Offline mode turned on by the -o command line option is honored from
the beginning even though command line options are applied to the
configuration last. If offline mode is not requested by the command line, it
may be switched on by either the user’s or the project’s config root.

If, on the other hand, the extends caches are specified in files that get
extended themselves, they won’t be used for assembling the configuration they
belong to (user’s or project’s, resp.). The extends cache specified by the
user’s defaults does, however, apply to downloading project configuration.
Let’s rewrite the config files, clean out the caches and re-run buildout:

While offline mode forbids network access completely, ‘newest’ mode determines
whether to look for updated versions of a resource even if some version of it
is already present locally. If we run buildout in newest mode
(newest = true), the configuration files are updated with each run:

Even when updating base configuration files with a buildout run, any given
configuration file will be downloaded only once during that particular run. If
some base configuration file is extended more than once, its cached copy is
used:

>>> zc.buildout.buildout.main([])
The URL http://localhost/baseA.cfg was downloaded.
The URL http://localhost/base.cfg was downloaded.
The URL http://localhost/baseB.cfg was downloaded.
Not upgrading because not running a local buildout command.
Unused options for buildout: 'bar' 'foo'.

The buildout section used to recognize an option named extended-by
that was deprecated at some point and removed in the 1.5 line. Since ignoring
this option silently was considered harmful as a matter of principle, a
UserError is raised if that option is encountered now:

zc buildout has a convenience command for running setup scripts. Why?
There are two reasons. If a setup script doesn’t import setuptools,
you can’t use any setuptools-provided commands, like bdist_egg. When
buildout runs a setup script, it arranges to import setuptools before
running the script so setuptools-provided commands are available.

If you use a squeaky-clean Python to do your development, the setup
script that would import setuptools because setuptools isn’t in the
path. Because buildout requires setuptools and knows where it has
installed a setuptools egg, it adds the setuptools egg to the Python
path before running the script. To run a setup script, use the
buildout setup command, passing the name of a script or a directory
containing a setup script and arguments to the script. Let’s look at
an example:

When a buildout is run, one of the first steps performed is to check for
updates to either zc.buildout or setuptools. To
demonstrate this, we’ve created some “new releases” of buildout and
setuptools in a new_releases folder:

When buildout restarts and the restarted buildout exits with an error code,
the original buildout that called the second buildout also exits with that
error code. Otherwise build scripts can erroneously detect a successful
buildout run even if it failed.

Buildouts can be pretty complex. When things go wrong, it isn’t
always obvious why. Errors can occur due to problems in user input or
due to bugs in zc.buildout or recipes. When an error occurs, Python’s
post-mortem debugger can be used to inspect the state of the buildout
or recipe code were there error occurred. To enable this, use the -D
option to the buildout. Let’s create a recipe that has a bug:

Buildout recipes provide reusable Python modules for common
configuration tasks. The most widely used recipes tend to provide
low-level functions, like installing eggs or software distributions,
creating configuration files, and so on. The normal recipe framework
is fairly well suited to building these general components.

Full-blown applications may require many, often tens, of parts.
Defining the many parts that make up an application can be tedious and
often entails a lot of repetition. Buildout provides a number of
mechanisms to avoid repetition, including merging of configuration
files and macros, but these, while useful to an extent, don’t scale
very well. Buildout isn’t and shouldn’t be a programming language.

Meta-recipes allow us to bring Python to bear to provide higher-level
abstractions for buildouts.

A meta-recipe is a regular Python recipe that primarily operates by
creating parts. A meta recipe isn’t merely a high level recipe. It’s
a recipe that defers most or all of it’s work to lower-level recipes by
manipulating the buildout database.

This buildout doesn’t build software. Rather it builds configuration
for deploying a database configuration using already-deployed
software. For the purpose of this document, however, the details are
totally unimportant.

Rather than crafting the configuration above every time, we can write
a meta-recipe that crafts it for us. We’ll use our meta-recipe as
follows:

The idea here is that the meta recipe allows us to specify the minimal
information necessary. A meta-recipe often automates policies and
assumptions that are application and organization dependent. The
example above assumes, for example, that we want to pack to 3
days in the past on Saturdays.

The meta recipe just adds parts to the buildout. It does this by
setting items and and calling the parse method. The parse
method just takes a string in buildout configuration syntax. It’s
useful when we want to add static, or nearly static part data. The
setting items syntax is useful when we have non-trivial computation
for part data.

The order that we add parts is important. When adding a part, any
string substitutions and other dependencies are evaluated, so the
referenced parts must be defined first. This is why, for example, the
pack part is added after the main part.

Note that the meta recipe supplied an integer for one of the
options. In addition to strings, it’s legal to supply integer values.

There are a few things to note about this example:

The install and update methods are empty.

While not required, this is a very common pattern for meta
recipes. Most meta recipes, simply invoke other recipes.

Setting a buildout item or calling parse, adds any sections with
recipes as parts.

The zc.buildout.testing module provides an API that can be used when
writing recipe tests. This API is documented below. Many examples of
using this API can be found in the zc.buildout, zc.recipe.egg, and
zc.recipe.testrunner tests.

The buildoutSetup function can be used as a doctest setup function.
It creates a sample buildout that can be used by tests, changing the
current working directory to the sample_buildout. It also adds a
number of names to the test namespace:

sample_buildout

This is the name of a buildout with a basic configuration.

buildout

This is the path of the buildout script in the sample buildout.

ls(*path)

List the contents of a directory. The directory path is provided as one or
more strings, to be joined with os.path.join.

cat(*path)

Display the contents of a file. The file path is provided as one or
more strings, to be joined with os.path.join.

On Windows, if the file doesn’t exist, the function will try
adding a ‘-script.py’ suffix. This helps to work around a
difference in script generation on windows.

mkdir(*path)

Create a directory. The directory path is provided as one or
more strings, to be joined with os.path.join.

rmdir(*path)

Remove a directory. The directory path is provided as one or
more strings, to be joined with os.path.join.

remove(*path)

Remove a directory or file. The path is provided as one or
more strings, to be joined with os.path.join.

tmpdir(name)

Create a temporary directory with the given name. The directory
will be automatically removed at the end of the test. The path of
the created directory is returned.

Further, if the the normalize_path normalizing substitution (see
below) is used, then any paths starting with this path will be
normalized to:

/name/restofpath

No two temporary directories can be created with the same name. A
directory created with tmpdir can be removed with rmdir and recreated.

Note that the sample_buildout directory is created by calling this
function.

write(*path_and_contents)

Create a file. The file path is provided as one or more strings,
to be joined with os.path.join. The last argument is the file contents.

system(command, input='')

Execute a system command with the given input passed to the
command’s standard input. The output (error and regular output)
from the command is returned.

get(url)

Get a web page.

cd(*path)

Change to the given directory. The directory path is provided as one or
more strings, to be joined with os.path.join.

The directory will be reset at the end of the test.

uncd()

Change to the directory that was current prior to the previous
call to cd. You can call cd multiple times and then
uncd the same number of times to return to the same location.

join(*path)

A convenient reference to os.path.join.

register_teardown(func)

Register a tear-down function. The function will be called with
no arguments at the end of the test.

start_server(path)

Start a web server on the given path. The server will be shut
down at the end of the test. The server URL is returned.

You can cause the server to start and stop logging it’s output
using:

>>> get(server_url+'enable_server_logging')

and:

>>> get(server_url+'disable_server_logging')

This can be useful to see how buildout is interacting with a
server.

sdist(setup, dest)

Create a source distribution by running the given setup file and
placing the result in the given destination directory. If the
setup argument is a directory, the setup.py file in that
directory is used.

bdist_egg(setup, dest)

Create an egg by running the given setup file
and placing the result in the given destination
directory. If the setup argument is a directory, then the
setup.py file in that directory is used.

Install eggs for a given project into a destination. If the
destination is a test object, then the eggs directory of the
sample buildout (sample_buildout) defined by the test will be used.
Tests will use this to install the distributions for the packages
being tested (and their dependencies) into a sample buildout. The egg
to be used should already be loaded, by importing one of the modules
provided, before calling this function.

Recipe tests often generate output that is dependent on temporary file
locations, operating system conventions or Python versions. To deal
with these dependencies, we often use
zope.testing.renormalizing.RENormalizing to normalize test output.
zope.testing.renormalizing.RENormalizing takes pairs of regular
expressions and substitutions. The zc.buildout.testing module provides
a few helpful variables that define regular-expression/substitution
pairs that you can pass to zope.testing.renormalizing.RENormalizing.

normalize_path

Converts tests paths, based on directories created with tmpdir(),
to simple paths.

normalize_script

On Unix-like systems, scripts are implemented in single files
without suffixes. On windows, scripts are implemented with 2
files, a -script.py file and a .exe file. This normalization
converts directory listings of Windows scripts to the form
generated on UNix-like systems.

normalize_egg_py

Normalize Python version and platform indicators, if specified, in
egg names.

The easy_install module provides some functions to provide support for
egg and script installation. It provides functionality at the python
level that is similar to easy_install, with a few exceptions:

By default, we look for new packages and the packages that
they depend on. This is somewhat like (and uses) the –upgrade
option of easy_install, except that we also upgrade required
packages.

If the highest-revision package satisfying a specification is
already present, then we don’t try to get another one. This saves a
lot of search time in the common case that packages are pegged to
specific versions.

If there is a develop egg that satisfies a requirement, we don’t
look for additional distributions. We always give preference to
develop eggs.

The easy_install module provides a function, install, for installing one
or more packages and their dependencies. The install function takes 2
positional arguments:

An iterable of setuptools requirement strings for the distributions
to be installed, and

A destination directory to install to and to satisfy requirements
from. The destination directory can be None, in which case, no new
distributions are downloaded and there will be an error if the
needed distributions can’t be found among those already installed.

It supports a number of optional keyword arguments:

links

A sequence of URLs, file names, or directories to look for
links to distributions.

index

The URL of an index server, or almost any other valid URL. :)

If not specified, the Python Package Index,
http://pypi.python.org/simple/, is used. You can specify an
alternate index with this option. If you use the links option and
if the links point to the needed distributions, then the index can
be anything and will be largely ignored. In the examples, here,
we’ll just point to an empty directory on our link server. This
will make our examples run a little bit faster.

path

A list of additional directories to search for locally-installed
distributions.

working_set

An existing working set to be augmented with additional
distributions, if necessary to satisfy requirements. This allows
you to call install multiple times, if necessary, to gather
multiple sets of requirements.

newest

A boolean value indicating whether to search for new distributions
when already-installed distributions meet the requirement. When
this is true, the default, and when the destination directory is
not None, then the install function will search for the newest
distributions that satisfy the requirements.

versions

A dictionary mapping project names to version numbers to be used
when selecting distributions. This can be used to specify a set of
distribution versions independent of other requirements.

use_dependency_links

A flag indicating whether to search for dependencies using the
setup dependency_links metadata or not. If true, links are searched
for using dependency_links in preference to other
locations. Defaults to true.

relative_paths

Adjust egg paths so they are relative to the script path. This
allows scripts to work when scripts and eggs are moved, as long as
they are both moved in the same way.

The install method returns a working set containing the distributions
needed to meet the given requirements.

We requested version 0.2 of the demo distribution to be installed into
the destination server. We specified that we should search for links
on the link server and that we should use the (empty) link server
index directory as a package index.

The working set contains the distributions we retrieved.

>>> for dist in ws:
... print_(dist)
demo 0.2
demoneeded 1.1

We got demoneeded because it was a dependency of demo.

And the actual eggs were added to the eggs directory.

>>> ls(dest)
d demo-0.2-py2.4.egg
d demoneeded-1.1-py2.4.egg

If we remove the version restriction on demo, but specify a false
value for newest, no new distributions will be installed:

Note that we didn’t get the newest versions available. There were
release candidates for newer versions of both packages. By default,
final releases are preferred. We can change this behavior using the
prefer_final function:

Sometimes it’s useful to specify version information independent of
normal requirements specifications. For example, a buildout may need
to lock down a set of versions, without having to put put version
numbers in setup files or part definitions. If a dictionary is passed
to the install function, mapping project names to version numbers,
then the versions numbers will be used.

Setuptools allows metadata that describes where to search for package
dependencies. This option is called dependency_links. Buildout has its
own notion of where to look for dependencies, but it also uses the
setup tools dependency_links information if it’s available.

Let’s demo this by creating an egg that specifies dependency_links.

To begin, let’s create a new egg repository. This repository hold a
newer version of the ‘demoneeded’ egg than the sample repository does.

Once again the dependency is fetched from the logging server even
though it is also available from the non-logging server. This is
because the version on the logging server is newer and buildout
normally chooses the newest egg available.

If you wish to control where dependencies come from regardless of
dependency_links setup metadata use the ‘use_dependency_links’ option
to zc.buildout.easy_install.install().

The easy_install module provides support for creating scripts from
eggs. It provides a function similar to setuptools except that it
provides facilities for baking a script’s path into the script. This
has two advantages:

The eggs to be used by a script are not chosen at run time, making
startup faster and, more importantly, deterministic.

The script doesn’t have to import pkg_resources because the logic
that pkg_resources would execute at run time is executed at
script-creation time.

The scripts method can be used to generate scripts. Let’s create a
destination directory for it to place them in:

>>> import tempfile
>>> bin = tmpdir('bin')

Now, we’ll use the scripts method to generate scripts in this directory
from the demo egg:

Note that in Windows, 2 files are generated for each script. A script
file, ending in ‘-script.py’, and an exe file that allows the script
to be invoked directly without having to specify the Python
interpreter and without having to provide a ‘.py’ suffix.

Passing entry-point information directly is handy when using eggs (or
distributions) that don’t declare their entry points, such as
distributions that aren’t based on setuptools.

The interpreter keyword argument can be used to generate a script that can
be used to invoke the Python interactive interpreter with the path set
based on the working set. This generated script can also be used to
run other scripts with the path set on the working set:

Sometimes, you want to be able to move a buildout directory around and
have scripts still work without having to rebuild them. We can
control this using the relative_paths option to install. You need
to pass a common base directory of the scripts and eggs:

Distutils copies the script files verbatim, apart from a line at the top that
looks like #!/usr/bin/python, which gets replaced by the actual python
interpreter. Buildout does the same, but additionally also adds the sys.path
like for the console_scripts.

Note that there are several items that need to come first in such a script
before buildout’s sys.path statements: a source encoding hint, a module
docstring and __future__ imports. Buildout retains them in their proper
place by looking at the first non-future import and placing its sys.path
statement before that.

Due to the nature of distutils scripts, buildout cannot pass arguments as
there’s no specific method to pass them to.

In some cases, a python 3 __pycache__ directory can end up in an internal
EGG-INFO metadata directory, next to the script information we’re looking
for. Buildout doesn’t crash on that:

Sometimes, we need to control how extension modules are built. The
build function provides this level of control. It takes a single
package specification, downloads a source distribution, and builds it
with specified custom build options.

The build function takes 3 positional arguments:

spec

A package specification for a source distribution

dest

A destination directory

build_ext

A dictionary of options to be passed to the distutils build_ext
command when building extensions.

It supports a number of optional keyword arguments:

links

a sequence of URLs, file names, or directories to look for
links to distributions,

index

The URL of an index server, or almost any other valid URL. :)

If not specified, the Python Package Index,
http://pypi.python.org/simple/, is used. You can specify an
alternate index with this option. If you use the links option and
if the links point to the needed distributions, then the index can
be anything and will be largely ignored. In the examples, here,
we’ll just point to an empty directory on our link server. This
will make our examples run a little bit faster.

path

A list of additional directories to search for locally-installed
distributions.

newest

A boolean value indicating whether to search for new distributions
when already-installed distributions meet the requirement. When
this is true, the default, and when the destination directory is
not None, then the install function will search for the newest
distributions that satisfy the requirements.

versions

A dictionary mapping project names to version numbers to be used
when selecting distributions. This can be used to specify a set of
distribution versions independent of other requirements.

Our link server included a source distribution that includes a simple
extension, extdemo.c:

The versions option also influences the versions used. For example,
if we specify a version for extdemo, then that will be used, even
though it isn’t the newest. Let’s clean out the destination directory
first:

Normally, when distributions are installed, if any processing is
needed, they are downloaded from the internet to a temporary directory
and then installed from there. A download cache can be used to avoid
the download step. This can be useful to reduce network access and to
create source distributions of an entire buildout.

A download cache is specified by calling the download_cache
function. The function always returns the previous setting. If no
argument is passed, then the setting is unchanged. If an argument is
passed, the download cache is set to the given path, which must point
to an existing directory. Passing None clears the cache setting.

To see this work, we’ll create a directory and set it as the cache
directory:

A download cache can be used as the basis of application source
releases. In an application source release, we want to distribute an
application that can be built without making any network accesses. In
this case, we distribute a download cache and tell the easy_install
module to install from the download cache only, without making network
accesses. The install_from_cache function can be used to signal that
packages should be installed only from the download cache. The
function always returns the previous setting. Calling it with no
arguments returns the current setting without changing it:

>>> zc.buildout.easy_install.install_from_cache()
False

Calling it with a boolean value changes the setting and returns the
previous setting:

Integrated the buildout-versions extension into buildout
itself. For this, a few options were added to buildout:

If show-picked-versions is set to true, all picked versions are
printed at the end of the buildout run. This saves you from running
buildout in verbose mode and extracting the picked versions from the
output.

If update-versions-file is set to a filename (relative to the buildout
directory), the show-picked-versions output is appended to that file.

Buildout options can be given on the command line using the form:

option_name=value

as a short-hand for:

buildout:option_name=value

The versions option now defaults to versions, so you no
longer need to include:

versions = versions

in a buildout section when pinning versions.

A versions section is added, if necessary, if a versions
option isn’t used.

Buildout-defined default versions are included in the versions
section, if there is one.

The buildout:zc.buildout-version and
buildout:distribute-version options have been removed in favor
of providing version constraints in a versions section.

Error if install-from-cache and offline are used together, because
offline largely means “don’t install”.

Provide better error messages when distributions can’t be installed
because buildout is run in offline mode.

Versions in versions sections can now be simple constraints, like
>=2.0dev in addition to being simple versions.

Buildout 2 leverages this to make sure it uses
zc.recipe.egg>=2.0.0a3, which mainly matters for Python 3.

The buildout init command now accepts distribution requirements and
paths to set up a custom interpreter part that has the distributions
or parts in the path. For example:

python bootstrap.py init BeautifulSoup

Added buildout:socket-timeout option so that socket timeout can be configured
both from command line and from config files. (gotcha)

Avoid sorting the working set and requirements when it won’t be
logged. When profiling a simple buildout with 10 parts with
identical and large working sets, this resulted in a decrease of run
time from 93.411 to 15.068 seconds, about a 6 fold improvement. To
see the benefit be sure to run without any increase in verbosity
(“-v” option). (rossp)

Introduce a cache for the expensive buildout._dir_hash function.

Remove duplicate path from script’s sys.path setup.

Make sure to download extended configuration files only once per buildout
run even if they are referenced multiple times (patch by Rafael Monnerat).

Removed any traces of the implementation of extended-by. Raise a
UserError if the option is encountered instead of ignoring it, though.

Using pre-detected setuptools version for easy_installing tgz files. This
prevents a recursion error when easy_installing an upgraded “distribute”
tgz. Note that setuptools did not have this recursion problem solely
because it was packaged as an .egg, which does not have to go through
the easy_install step.

When doing variable substitutions, you can omit the section name to
refer to a variable in the same section (e.g. ${:foo}).

When doing variable substitution, you can use the special option,
_buildout_section_name_ to get the section name. This is most handy
for getting the current section name (e.g. ${:_buildout_section_name_}).

Now the bootstrap.py has an optional –version argument,
that can be used to force buildout version to use.

zc.buildout.testing.buildoutSetUp installs a new handler in the
python root logging facility. This handler is now removed during
tear down as it might disturb other packages reusing buildout’s
testing infrastructure.

fixed usage of ‘relative_paths’ keyword parameter on Windows

Added an unload entry point for extensions.

Fixed bug: when the relative paths option was used, relative paths
could be inserted into sys.path if a relative path was used to run
the generated script.

fixed the test for the += -= syntax in buildout.txt as the test
was actually wrong. The original implementation did a split/join
on whitespace, and later on that was corrected to respect the original
EOL setting, the test was not updated, though. (seletz)

further fixes for “”“AttributeError: Buildout instance has no
attribute ‘_logger’”“” by providing reasonable defaults
within the Buildout constructor (related to the new ‘allow-hosts’ option)
(patch by Gottfried Ganssauge) (ajung)

Added a configuration option that allows buildouts to ignore
dependency_links metadata specified in setup. By default
dependency_links in setup are used in addition to buildout specified
find-links. This can make it hard to control where eggs come
from. Here’s how to tell buildout to ignore URLs in
dependency_links:

[buildout]
use-dependency-links = false

By default use-dependency-links is true, which matches the behavior
of previous versions of buildout.

Added a configuration option that causes buildout to error if a
version is picked. This is a nice safety belt when fixing all
versions is intended, especially when creating releases.

151820: Develop failed if the setup.py script imported modules in
the distribution directory.

Verbose logging of the develop command was omitting detailed
output.

The setup command wasn’t documented.

The setup command failed if run in a directory without specifying a
configuration file.

The setup command raised a stupid exception if run without arguments.

When using a local find links or index, distributions weren’t copied
to the download cache.

When installing from source releases, a version specification (via a
buildout versions section) for setuptools was ignored when deciding
which setuptools to use to build an egg from the source release.

Now, final distributions are preferred over non-final versions. If
both final and non-final versions satisfy a requirement, then the
final version will be used even if it is older. The normal way to
override this for specific packages is to specifically require a
non-final version, either specifically or via a lower bound.

There is a buildout prefer-final version that can be used with a
value of “false”:

prefer-final = false

To prefer newer versions, regardless of whether or not they are
final, buildout-wide.

The new simple Python index, http://cheeseshop.python.org/simple, is
used as the default index. This will provide better performance
than the human package index interface,
http://pypi.python.org/pypi. More importantly, it lists hidden
distributions, so buildouts with fixed distribution versions will be
able to find old distributions even if the distributions have been
hidden in the human PyPI interface.

Bugs Fixed

Feature Changes

Thanks to recent fixes in setuptools, I was able to change buildout
to use find-link and index information when searching extensions.

Sadly, this work, especially the timing, was motivated my the need
to use alternate indexes due to performance problems in the cheese
shop (http://www.python.org/pypi/). I really home we can address
these performance problems soon.

Added support for download caches. A buildout can specify a cache
for distribution downloads. The cache can be shared among buildouts
to reduce network access and to support creating source
distributions for applications allowing install without network
access.

Feature Changes

Added support for repeatable buildouts by allowing egg versions to
be specified in a versions section.

The easy_install module install and build functions now accept a
versions argument that supplied to mapping from project name to
version numbers. This can be used to fix version numbers for
required distributions and their dependencies.

When a version isn’t fixed, using either a versions option or using
a fixed version number in a requirement, then a debug log message is
emitted indicating the version picked. This is useful for setting
versions options.

A default_versions function can be used to set a default value for
this option.

Adjusted the output for verbosity levels. Using a single -v option
no longer causes voluminous setuptools output. Using -vv and -vvv
now triggers extra setuptools output.

Added a buildout newest option, to control whether the newest
distributions should be sought to meet requirements. This might
also provide a hint to recipes that don’t deal with
distributions. For example, a recipe that manages subversion
checkouts might not update a checkout if newest is set to “false”.

Added a newest keyword parameter to the
zc.buildout.easy_install.install and zc.buildout.easy_install.build
functions to control whether the newest distributions that need
given requirements should be sought. If a false value is provided
for this parameter and already installed eggs meet the given
requirements, then no attempt will be made to search for newer
distributions.

The recipe-testing support setUp function now adds the name
buildout to the test namespace with a value that is the path to
the buildout script in the sample buildout. This allows tests to
use

A part referenced by a part in a parts list is now added to the parts
list before the referencing part. This means that you can omit
parts from the parts list if they are referenced by other parts.

Added a develop function to the easy_install module to aid in
creating develop eggs with custom build_ext options.

The build and develop functions in the easy_install module now
return the path of the egg or egg link created.

Removed the limitation that parts named in the install command can
only name configured parts.

Removed support ConfigParser-style variable substitutions
(e.g. %(foo)s). Only the string-template style of variable
(e.g. ${section:option}) substitutions will be supported.
Supporting both violates “there’s only one way to do it”.

We treat setuptools as a dependency of any distribution that
(declares that it) uses namespace packages, whether it declares
setuptools as a dependency or not. This wasn’t working for eggs
installed by virtue of being dependencies.

Added a recipe update method. Now install is only called when a part
is installed for the first time, or after an uninstall. Otherwise,
update is called. For backward compatibility, recipes that don’t
define update methods are still supported.

If a distribution defines namespace packages but fails to declare
setuptools as one of its dependencies, we now treat setuptools as an
implicit dependency. We generate a warning if the distribution
is a develop egg.

You can now create develop eggs for setup scripts that don’t use setuptools.

Refactored to do more work in buildout and less work in easy_install.
This makes things go a little faster, makes errors a little easier to
handle, and allows extensions (like the sftp extension) to influence
more of the process. This was done to fix a problem in using the sftp
support.

Changed the bootstrapping code to only install setuptools and
buildout. The bootstrap code no-longer runs the buildout itself.
This was to fix a bug that caused parts to be recreated
unnecessarily because the recipe signature in the initial buildout
reflected temporary locations for setuptools and buildout.

Now create a minimal setup.py if it doesn’t exist and issue a
warning that it is being created.