Cloud Services

Congress will work with any cloud service, as long as Congress can
represent the service’s state in table format. A table is a
collection of rows, where each row is a collection of columns, and
each row-column entry contains a string or a number.

For example, Neutron contains a mapping between IP addresses and the
ports they are assigned to; neutron represents this state as the
following table.:

To plug a new service into Congress, you write a small piece of code,
called a driver, that queries the new service (usually through API calls)
and translates the service state into tables of data. Out of the box
Congress includes drivers for a number of common services (see below).

For example, the driver for Neutron invokes the Neutron API calls that list
networks, ports, security groups, and routers. The driver translates each of
the JSON objects that the API calls return into tables (where in Python a table
is a list of tuples). The Neutron driver is implemented here:

congress/datasources/neutronv2_driver.py

Once the driver is available, you install it into Congress,
you configure it (such as with an IP address/port/username), and you
write policy that references the tables populated by that driver.

To install a new driver, you must add its location to the Congress
configuration file and restart the server. Congress has a single
configuration parameter (called drivers) that is a list of all the
installed drivers. To install a new driver, simply add to this list
and restart.

For example, to install the Neutron driver, you add the following to the
list of drivers in the configuration file:

Once the driver code is in place, you can use it to create a datasource whose
data is available to Congress policies. To create a datasource, you use the API and
provide a unique name (the name you will use in policy to refer to the service), the
name of the datasource driver you want to use, and additional connection details
needed by your service (such as an IP and a username/password).

For example, using the Congress CLI, you can create a datasource named ‘neutron_test’ using the
‘neutronv2’ driver:

And if you had a second instance of Neutron running to manage
your production network, you could create a second datasource (named say ‘neutron_prod’)
using the neutronv2 driver so that you could write policy over both instances of Neutron.

When you write policy, you would use the name ‘neutron_test:ports’ to reference the ‘ports’
table generated by the ‘neutron_test’ datasource, and you use ‘neutron_test:networks’ to
reference the ‘networks’ table generated by the ‘neutron_test’ datasource. Similarly,
you use ‘neutron_prod:ports’ and ‘neutron_prod:networks’ to reference the
tables populated by the ‘neutron_prod’ datasource.
(More details about writing policy can be found in the
Policy section.)

Congress currently has drivers for each of the following services. Each driver
has a differing degree of coverage for the available API calls.

OpenStack Aodh

OpenStack Cinder

OpenStack Glance (v2)

OpenStack Heat

OpenStack Ironic

OpenStack Keystone (v2 & v3)

OpenStack Mistral

OpenStack Monasca

OpenStack Murano

OpenStack Neutron (v2)

OpenStack Neutron QoS

OpenStack Nova

OpenStack Swift

Cloud Foundry

Plexxi

vCenter

OPNFV Doctor

Using the API or CLI, you can review the list of tables and columns that a driver supports.
Roughly, you can think of each table as a collection of objects (like networks or servers),
and the columns of that table as the attributes of those objects (like name, status, or ID).
The value of each row-column entry is a (Python) string or number. If
the attribute as returned by the API call is a complex object, that object
is flattened into its own table (or tables).

Typically, you will create a subclass of
datasource_driver.PollingDataSourceDriver or
datasource_driver.PushedDataSourceDriver depending on the type of your
datasource driver. Each instance of that class will correspond to a different
service using that driver.

The following steps detail how to implement a polling datasource driver.

This function is called to update self.state to reflect the new
state of the service. self.state is a dictionary that maps a
tablename (as a string) to a set of tuples (to a collection of tables).
Each tuple element must be either a number or string. This function
implements the polling logic for the service.

5. By convention, it is useful for debugging purposes to include a
main that calls update_from_datasource, and prints out the raw
API results along with the tables that were generated.

To install and test the newly written driver, please follow the new driver
installation procedure mentioned in :ref: Driver installation <driver-installation>
section.

Since Congress requires the state of each dataservice to be represented as
tables, we must convert the results of each API call (which may be comprised
of dictionaries, lists, dictionaries embedded within lists, etc.) into tables.

Congress provides a translation method to make the translation from API
results into tables convenient. The translation method takes a description of
the API data structure, and converts objects of that structure into rows of
one or more tables (depending on the data structure). For example, this is a
partial snippet from the Neutron driver:

This networks_translator describes a python dictionary data structure that
contains four keys: id, name, tenant_id, and subnets. The value for the
subnets key is a list of subnet_group_ids each of which is a number. For
example:

{ “id”: 1234,

“name”: “Network Foo”,
“tenant_id”: 5678,
“subnets”: [ 100, 101 ] }

Given the networks_translator description, the translator creates two tables.
The first table is named “networks” with a column for name, subnets,
tenant_id, and id. The second table will be named “networks.subnet” and will
contain two columns, one containing the subnet_group_id, and the second
containing an ID that associates the row in the network to the rows in the
networks.subnets table.

To use the translation methods, the driver defines a translator such as
networks_translator and then passes the API response objects to
translate_objs() which is defined in congress/datasources/datasource_driver.py
See congress/datasources/neutron_driver.py as an example.

The convenience translators may be insufficient in some cases, for example,
the data source may provide data in an unusual format, the convenience
translators may be inefficient, or the fixed translation method may result in
an unsuitable table schema. In such cases, a driver may need to implement its
own translation. In those cases, we have a few recommendations.

Recommendation 1: Row = object. Typically an API call will return a
collection of objects (e.g. networks, virtual machines, disks). Conceptually
it is convenient to represent each object with a row in a table. The columns
of that row are the attributes of each object. For example, a table of all
virtual machines will have columns for memory, disk, flavor, and image.

Table: virtual_machine

ID

Memory

Disk

Flavor

Image

66dafde0-a49c-11e3-be40-425861b86ab6

256GB

1TB

1

83e31d4c-a49c-11e3-be40-425861b86ab6

73e31d4c-a49c-11e3-be40-425861b86ab6

10GB

2TB

2

93e31d4c-a49c-11e3-be40-425861b86ab6

Recommendation 2. Avoid wide tables. Wide tables (i.e. tables with many
columns) are hard to use for a policy-writer. Breaking such tables up into
smaller ones is often a good idea. In the above example, we could create 4
tables with 2 columns instead of 1 table with 5 columns.

Table: virtual_machine.memory

ID

Memory

66dafde0-a49c-11e3-be40-425861b86ab6

256GB

73e31d4c-a49c-11e3-be40-425861b86ab6

10GB

Table: virtual_machine.disk

ID

Disk

66dafde0-a49c-11e3-be40-425861b86ab6

1TB

73e31d4c-a49c-11e3-be40-425861b86ab6

2TB

Table: virtual_machine.flavor

ID

Flavor

66dafde0-a49c-11e3-be40-425861b86ab6

1

73e31d4c-a49c-11e3-be40-425861b86ab6

2

Table: virtual_machine.image

ID

Image

66dafde0-a49c-11e3-be40-425861b86ab6

83e31d4c-a49c-11e3-be40-425861b86ab6

73e31d4c-a49c-11e3-be40-425861b86ab6

93e31d4c-a49c-11e3-be40-425861b86ab6

Recommendation 3. Try these design patterns. Below we give a few design
patterns. Notice that when an object has an attribute whose value is a
structured object itself (e.g. a list of dictionaries), we must recursively
flatten that subobject into tables.

Next the test creates a fake Glance client. Glance is an OpenStack service that stores (among other things) operating system Images that you can use to create a new VM. The Glance datasource driver makes a call to <glance-client>.images.list() to retrieve the list of those images, and then turns that list of images into tables. The test creates a fake Glance client so it can control the return value of <glance-client>.images.list().

Next the test instantiates the GlanceV2Driver class, which contains the code for the Glance driver. Passing ‘poll_time’ as 0 is probably unnecessary here, but it tells the driver not to poll automatically. Passing ‘client’ is important because it tells the GlanceV2Driver class to use a mocked version of the Glance client instead of creating its own.

Next the test defines which value it wants <glance-client>.images.list() to return. The test itself will check if the Glance driver code properly translates this return value into tables. So this is the actual input to the test. Either you can write this by hand, or you can run the heat-client and print out the results.

test_update_from_datasource() is the actual test, where we have the datasource driver grab the list of Glance images and translate them to tables. The test runs the update_from_datasource() method like normal except it ensures the return value of <glance-client>.images.list() is self.mock_images.

deftest_update_from_datasource(self):

The first thing the method does is set the return value of self.driver.glance.images.list() to self.mock_images[‘images’]. Then it calls update_from_datasource() in the usual way, which translates self.mock_images[‘images’] into tables and stores the result into the driver’s self.state dictionary.

Next the test defines the tables that update_from_datasource() should construct. Actually, the test defines the expected value of Glance’s self.state when update_from_datasource() finishes. Remember that self.state is a dictionary mapping a table name to the set of tuples that belong to the table. For Glance, there’s just one table: ‘images’, and so the expected self.state is a dictionary with one key ‘images’ and one value: a set of tuples.