Navigation

The data layer combines access to stored state and messages, ensuring consistency between them, and exposing a well-defined API that can be used both internally and externally.
Using caching and the clock information provided by the db and mq layers, this layer ensures that its callers can easily receive a dump of current state plus changes to that state, without missing or duplicating messages.

control - allows state to be changed in specific ways by sending appropriate messages (e.g., stopping a build); and

updates - direct updates to state appropriate messages.

The getters and subscriptions are exposed everywhere.
Access to the control section should be authenticated at higher levels, as the data layer does no authentication.
The updates section is for use only by the process layer.

The interfaces for all sections but the updates sections are intended to be language-agnostic.
That is, they should be callable from JavaScript via HTTP, or via some other interface added to Buildbot after the fact.

The getter section can get either a single resource, or a list of resources.
Getting a single resource requires a resource identifier (a tuple of strings) and a set of options to support automatic expansion of links to other resources (thus saving round-trips).
Lists are requested with a partial resource identifier (a tuple of strings) and an optional set of filter options.
In some cases, certain filters are implicit in the path, e.g., the list of buildsteps for a particular build.

Message subscriptions can be made to anything that can be listed or gotten from the getter sections, using the same resource identifiers.
Options and explicit filters are not supported - a message contains only the most basic information about a resource, and a list subscription results in a message for every new resource of the desired type.
Implicit filters are supported.

The control sections defines a set of actions that cause Buildbot to behave in a certain way, e.g., rebuilding a build or shutting down a worker.
Actions correspond to a particular resource, although sometimes that resource is the root resource (an empty tuple).

The updates section defines a free-form set of methods that Buildbot's process implementation calls to update data.
Most update methods both modify state via the db API and send a message via the mq API.
Some are simple wrappers for these APIs, while others contain more complex logic, e.g., building a source stamp set for a collection of changes.
This section is the proper place to put common functionality, e.g., rebuilding builds or assembling buildsets.

This class implements the root of the data API.
Within the buildmaster process, the data connector is available at self.master.data.
The first three sections are implemented with the get and control methods, respectively, while the updates section is implemented using the updates attribute.
The path arguments to these methods should always be tuples.
Integer arguments can be presented as either integers or strings that can be parsed by int; all other arguments must be strings.

Get the endpoint responsible for the given path, along with any arguments extracted from the path.
This can be used by callers that need access to information from the endpoint beyond that returned from get.

This method implements the production of an event, for the rtype identified by its name string.
Usually, this is the role of the data layer to produce the events inside the update methods.
For the potential use cases where it would make sense to solely produce an event, and not update data, please use this API, rather than directly call mq.
It ensures the event is sent to all the routingkeys specified by eventPathPatterns.

This object has an attribute named for each resource type, named after the singular form (e.g., self.master.data.builder).
These attributes allow resource types to access one another for purposes of coordination.
They are not intended for external access -- all external access to the data API should be via the methods above or update methods.

The data API may be extended in various ways: adding new endpoints, new fields to resource types, new update methods, or entirely new resource types.
In any case, you should only extend the API if you plan to submit the extensions to be merged into Buildbot itself.
Private API extensions are strongly discouraged.

You'll need to use both plural and singular forms of the resource type; in this example, we'll use 'pub' and 'pubs'.
You can also follow an existing file, like master/buildbot/data/changes.py, to see when to use which form.

Each resource path is implemented as an Endpoint instance.
In most cases, each instance is of a different class, but this is not required.

The data connector's get and control methods both take a path argument that is used to look up the corresponding endpoint.
The path matching is performed by buildbot.util.pathmatch, and supports automatically extracting variable fields from the path.
See that module's description for details.

This attribute defines the path patterns which incoming paths must match to select this endpoint.
Paths are specified as URIs, and can contain variables as parsed by buildbot.util.pathmatch.Matcher.
Multiple paths are separated by whitespace.

For example, the following specifies two paths with the second having a single variable:

Raw resources are used to get the data not encoded in JSON via the rest API.
In the REST principles, this should be done via another endpoint, and not via a query parameter.
The get() method from endpoint should return following data structure:

{"raw":u"raw data to be sent to the http client","mime-type":u"<mime-type>","filename":u"filename_to_be_used_in_content_disposition_attachement_header"}

Get data from the endpoint.
This should return either a list of dictionaries (for list endpoints), a dictionary, or None (both for details endpoints).
The endpoint is free to handle any part of the result spec.
When doing so, it should remove the relevant configuration from the spec.
See below.

Any result spec configuration that remains on return will be applied automatically.

Endpoint implementations must have unit tests.
An endpoint's path should be documented in the .rst file for its resource type.

The initial pass at implementing any endpoint should just ignore the resultSpec argument to get.
After that initial pass, the argument can be used to optimize certain types of queries.
For example, if the resource type has many resources, but most real-life queries use the result spec to filter out all but a few resources from that group, then it makes sense for the endpoint to examine the result spec and allow the underlying DB API to do that filtering.

When an endpoint handles parts of the result spec, it must remove those parts from the spec before it returns.
See the documentation for ResultSpec for methods to do so.

Note that endpoints must be careful not to alter the order of the filtering applied for a result spec.
For example, if an endpoint implements pagination, then it must also completely implement filtering and ordering, since those operations precede pagination in the result spec application.

Message types are defined in master/buildbot/test/util/validation.py, via the message module-level value.
This is a dictionary of MessageValidator objects, one for each message type.
The message type is determined from the first atom of its routing key.
The events dictionary lists the possible last atoms of the routing key.
It should be identical to the attribute of the ResourceType with the same name.

Update methods are for use by the Buildbot process code, and as such are generally designed to suit the needs of that code.
They generally encapsulate logic common to multiple users (e.g., creating buildsets), and finish by performing modifications in the database and sending a corresponding message.
In general, Buildbot does not depend on timing of either the database or message broker, so the order in which these operations are initiated is not important.

Update methods are considered part of Buildbot's user-visible interface, and as such incompatible changes should be avoided wherever possible.
Instead, either add a new method (and potentially re-implement existing methods in terms of the new method) or add new, optional parameters to an existing method.
If an incompatible change is unavoidable, it should be described clearly in the release notes.

Update methods are implemented as methods of ResourceType subclasses, decorated with @base.updateMethod:

The details of the fields of a resource type are rigorously enforced at several points in the Buildbot tests.
The enforcement is performed by the buildbot.data.types module.

The module provides a number of type classes for basic and compound types.
Each resource type class defines its entity type in its entityType class attribute.
Other resource types may refer to this class attribute if they embed an entity of that type.

The types are used both for tests, and by the REST interface to properly decode user-supplied query parameters.

A data resource is represented by a dictionary with well-known keys.
To define those keys and their values, subclass the Entity class within your ResourceType class and include each field as an attribute:

The data api enforces a strong and well-defined model on Buildbot's data.
This model is influenced by REST, in the sense that it defines resources, representations for resources, and identifiers for resources.
For each resource type, the API specifies

the attributes of the resource and their types (e.g., changes have a string specifying their project);

the format of links to other resources (e.g., buildsets to sourcestamp sets);

the paths relating to the resource type;

the format of routing keys for messages relating to the resource type;

the events that can occur on that resource (e.g., a buildrequest can be claimed); and

options and filters for getting resources.

Some resource type attributes only appear in certain formats, as noted in the documentation for the resource types.
In general, messages do not include any optional attributes, nor links.

Paths are given here separated by slashes, with key names prefixed by : and described below.
Similarly, message routing keys given here are separated by dots, with key names prefixed by $.
The translation to tuples and other formats should be obvious.

All strings in the data model are unicode strings.

[1]

The JSON API defined by status_json.py in Buildbot-0.8.x is considered version 1, although its root path was json, not api/v1.