mPlane Software Development Kit for Python 3

This module provides software development kit for building applications on top of the mPlane protocol. It is organized into serval modules:

mplane.model implements the mPlane protocol information model: message
types, the element registry, and various support classes. On top of the
information model, the mplane.scheduler module defines a framework for
binding mplane.model.Capability classes to runnable code, and for
invoking that code on the receipt of mPlane Statements; this is used to build
clients and components.

The mplane.client module defines interfaces for building clients; the mpcli script provides a simple command-line interface to this client.

The mplane.component module defines interfaces for building components; the component runtime can be started by running the mpcom script.

This software is copyright 2013-2015 the mPlane Consortium.
It is made available under the terms of the
GNU Lesser General Public License,
version 3 or, at your option, any later version.

This module implements Statements and Notifications, the core
messages used by the mPlane protocol to describe measurement
and query schemas, and various other classes to support them.

There are three kinds of Statements:

Capability represents something a component can do

Specification tells a component to do something it
advertised a Capability for

Result returns the results for a Specification in-band

Notifications are used to transfer other information between
components and clients. There are four kinds of Notifications:

Receipt notifies that a Result is not yet ready or
that the results of an operation will be indirectly exported.

Redemption is a subsequent attempt to redeem a Receipt.

Withdrawal notifies that a Capability is no longer available.

Interrupt notifies that a running Specification should be stopped.

To see how all this fits together, let’s simulate the message exchange
in a simple ping measurement. Initially, we have to load the default element
registry and programatically create a new empty Capability, as it would
be advertised by the component.

First, we set a temporal scope for the capability. Probe components
generally advertise a temporal scope from the present stretching
into the indeterminate future. In this case, we advertise that the
measurement performed is periodic, by setting the minimum period
supported by the capability: one ping per second.

>>> cap.set_when("now ... future / 1s")

We can only ping from one IPv4 address, to any IPv4 address.
Adding a parameter without a constraint makes it unconstrained:

Here we have a specification with a given token, schema, and 2 parameters,
no metadata, and five result columns.

Note

The schema of the statement is identified by a
schema hash, the first eight hex digits of which are shown for
diagnostic purposes. Statements with identical sets of parameters
and columns (schemas) will have identical schema hashes. Likewise,
the token is defined by the schema as well as the parameter values.

First, let’s fill in a specific temporal scope for the measurement:

>>> spec.set_when("2017-12-24 22:18:42 + 1m / 1s")

And then let’s fill in some parameters. All od the parameters whose
single values are already given by their constraints (in this case, source.ip4)
have already been filled in. So let’s start with the destination.
Note that strings are accepted and automatically parsed using each
parameter’s primitive type:

>>> spec.set_parameter_value("destination.ip4","10.0.37.2")

And now we can transform this specification and send it back to
the component from which we got the capability:

If the component cannot return results immediately (for example, because
the measurement will take some time), it can return a receipt instead:

>>> rcpt=mplane.model.Receipt(specification=comspec)

This receipt contains all the information in the specification, as well as a token
which can be used to quickly identify it in the future.

>>> rcpt.get_token()'e00d7fe813cf17eeeea37b313dcfa4e7'

Note

The mPlane protocol specification allows components to assign tokens
however they like. In the reference implementation, the default token
is based on a hash like the schema hash: statements with the same verb,
schema, parameter values, and metadata will have identical default tokens.
A component could, however, assign serial-number based tokens, or tokens
mapping to structures in its own filesystem, etc.

Notifications are used to send additional information between
mPlane clients and components other than measurement statements.
Notifications can either be part of a normal measurement workflow
(as Receipts and Redemptions) or signal exceptional conditions
(as Withdrawals and Interrupts).

This class contains implementation common to all Notifications
which do not contain any information from a related Capability
or Specification.

A Capability represents something an mPlane component can do.
Capabilities contain verbs (strings identifying the thing the
component can do), parameters (which must be given by a client
in a Specification in order for the component to do that thing),
metadata (additional information about the process used to do
that thing), and result columns (the data that thing will return).

Capabilities can either be created programatically, using the
add_parameter(), add_metadata(), and add_result_column()
methods, or by reading from a JSON object using parse_json().

Determines based on naming rules if this element is compatible with
element rval; that is, if transformation_to will return a function
for turning a value of this element to the other. Compatibility based
on name structure is a future feature; this method currently checks for
name equality only.

Returns a function which will transform values of this element
into values of element rval; used to support unit conversions.
This is a future feature, and is currently a no-op.
Only valid if compatible_with returns True.

A Parameter is an element which can take a constraint and a value.
In Capabilities, Parameters have constraints and no value; in
Specifications and Results, Parameters have both constraints and
values.

Returns True if the parameter can take the specified value,
False otherwise.
Either takes a list of values or a single value of the correct type for
the associated Primitive, or a list of strings or a single string, which
will be parsed to the correct type.

Sets the value of the Parameter.
Either takes a list of values or a single value of the correct type for
the associated Primitive, or a list of strings or a single string, which
will be parsed to the correct type.

A result is a statement that a component measured
a given set of values at a given point in time, according to a specification.

Results are generally created by passing the specification the new result responds to as the specification= argument to the constructor. A result inherits its token from the specification it responds to.

A ResultColumn is an element which can take an array of values.
In Capabilities and Specifications, this array is empty, while in
Results it has one or more values, such that all the ResultColumns
in the Result have the same number of values.

A Specification represents a request for an mPlane component to do
something it has advertised in a Capability.
Capabilities contain verbs (strings identifying the thing the
component can do), parameters (which must be given by a client
in a Specification in order for the component to do that thing),
metadata (additional information about the process used to do
that thing), and result columns (the data that thing will return).

Specifications are created either by passing a Capability the
Specification is intended to use as the capability= argument of
the constructor, or by reading from a JSON object (see model.parse_json()).

Returns True if this Specification fulfills the Capability
A specification fulfills a capability if the schemas match,
if the temporal scope of the specification is covered by that
of the capability, and if the specification’s parameter values
meet the capability’s parameter constraints.

Determine if a specification can be scheduled – i.e., that its
temporal scope refers to some time in the future at which a
measurement or other operation should take place, or some range of
time for which existing data should be searched and/or retrieved.

Iterates over subordinate specifications if this specification is repeated
(i.e., has a repeated Temporal Scope); otherwise yields self once. Each subordinate
specification has an absolute temporal scope derived from this specification’s
relative temporal scope and schedule.

Sets the statement’s temporal scope. Ensures that the temporal scope is
within the previous temporal scope unless force is True.
Takes either an instance of
mplane.model.When, or a string describing the scope.

Returns a tuple with delays for timers to signal the start and end of
a temporal scope, given a specified time zero, which defaults to the
current system time.

The start delay is defined to be zero if the scheduled start time has
already passed or the temporal scope is immediate (i.e., starts now).
The start delay is None if the temporal scope has expired (that is,
the current time is after the calculated end time).

The end delay is defined to be None if the temporal scope has already
expired, or if the temporal scope has no scheduled end (is infinite or
a singleton). End delays are calculated to give priority to duration
when a temporal scope is expressed in terms of duration, and to
prioritize end time otherwise.

Used in scheduling an enclosing Specification for execution.
Has no meaning for Capabilities or Results.

Transform an mPlane message into a JSON object representing it. If
token_only is True, uses tokens only for message types for which that is
appropriate (i.e. Reciepts, Redemptions, Withdrawals, and Interrupts).

A Job binds some running code to an mPlane.model.Specification
within a component. A Job can be thought of as a specific
instance of a Service presently running, or ready to run at some
point in the future.

Scheduler implements the common runtime of a Component within the
reference implementation. Components register Services bound to
Capabilities with add_service(), and submit jobs for scheduling using
submit_job().

Run this service given a specification which matches the capability.
This is called by the scheduler, and should be implemented by
a concrete subclass of Service.

The implementation should extract its parameters from a given
mplane.model.Specification, and return its result values in a
mplane.model.Result derived therefrom.

After each row or logically grouped set of rows, the implementation
should call the check_interrupt function to determine whether it
should stop; if this function returns True, the implementation should
terminate its processing in an orderly fashion and return its results.

Given a capability, return all known capabilities matching the
given schema capability. A capability matches a schema capability
if and only if: (1) the capability schemas match and (2) all
constraints in the capability are contained by all constraints
in the schema capability.

Used to programmatically select capabilities matching an
aggregation or other collection operation (e.g. at a supervisor).

Given a capability token or label, a temporal scope, a dictionary
of parameters, and an optional new label, derive a specification
and queue it for retrieval by the appropriate identity (i.e., the
one associated with the capability).

If the identity has indicated it supports callback control,
the optional callback_when parameter queues a callback spec to
schedule the next callback.

Receives mPlane messages POSTed from a client, and passes them to a
scheduler for processing. After waiting for a specified delay to see
if a Result is immediately available, returns a receipt for future
redemption.