README.md

This tutorial is an introduction to
FIWARE Cygnus - a generic
enabler which is used to persist context data into third-party databases using Apache Flume
creating a historical view of the context. The tutorial activates the IoT
sensors connected in the
previous tutorial and persists
measurements from those sensors into a database for further analysis.

Data Persistence

"History will be kind to me for I intend to write it."

— Winston Churchill

Previous tutorials have introduced a set of IoT Sensors (providing measurements of the state of the real world), and two
FIWARE Components - the Orion Context Broker and an IoT Agent. This tutorial will introduce a new data
persistence component - FIWARE Cygnus.

The system so far has been built up to handle the current context, in other words it holds the data entities defining
the state of the real-world objects at a given moment in time.

From this definition you can see - context is only interested in the current state of the system It is not the
responsibility of any of the existing components to report on the historical state of the system, the context is based
on the last measurement each sensor has sent to the context broker.

In order to do this, we will need to extend the existing architecture to persist changes of state into a database
whenever the context is updated.

Persisting historical context data is useful for big data analysis - it can be used to discover trends, or data can be
sampled and aggregated to remove the influence of outlying data measurements. However within each Smart Solution, the
significance of each entity type will differ and entities and attributes may need to be sampled at different rates.

Since the business requirements for using context data differ from application to application, there is no one standard
use case for historical data persistence - each situation is unique - it is not the case that one size fits all.
Therefore rather than overloading the context broker with the job of historical context data persistence, this role has
been separated out into a separate, highly configurable component - Cygnus.

As you would expect, Cygnus, as part of an Open Source platform, is technology agnostic regarding the database to be
used for data persistence. The database you choose to use will depend upon your own business needs.

However there is a cost to offering this flexibility - each part of the system must be separately configured and
notifications must be set up to only pass the minimal data required as necessary.

Device Monitor

For the purpose of this tutorial, a series of dummy IoT devices have been created, which will be attached to the context
broker. Details of the architecture and protocol used can be found in the
IoT Sensors tutorial. The state of each device can be seen on the
UltraLight device monitor web page found at: http://localhost:3000/device/monitor

Architecture

This application builds on the components and dummy IoT devices created in
previous tutorials. It will make use of three FIWARE components - the
Orion Context Broker, the
IoT Agent for Ultralight 2.0 and introduce the
Cygnus Generic Enabler for persisting context data to a database.
Additional databases are now involved - both the Orion Context Broker and the IoT Agent rely on
MongoDB technology to keep persistence of the information they hold, and we will be
persisting our historical context data another database - either MySQL , PostgreSQL or MongoDB database.

Therefore the overall architecture will consist of the following elements:

The FIWARE IoT Agent for Ultralight 2.0 which will
receive northbound measurements from the dummy IoT devices in
Ultralight 2.0
format and convert them to NGSI requests for the
context broker to alter the state of the context entities

FIWARE Cygnus which will subscribe to context changes and
persist them into a database (MySQL , PostgreSQL or MongoDB)

Docker Compose is a tool for defining and running multi-container Docker applications. A series of
YAML files are used configure the
required services for the application. This means all container services can be brought up in a single command. Docker
Compose is installed by default as part of Docker for Windows and Docker for Mac, however Linux users will need to
follow the instructions found here

You can check your current Docker and Docker Compose versions using the following commands:

docker-compose -vdocker version

Please ensure that you are using Docker version 18.03 or higher and Docker Compose 1.21 or higher and upgrade if
necessary.

Cygwin for Windows

We will start up our services using a simple Bash script. Windows users should download cygwin
to provide a command-line functionality similar to a Linux distribution on Windows.

Start Up

Before you start you should ensure that you have obtained or built the necessary Docker images locally. Please clone the
repository and create the necessary images by running the commands as shown:

Thereafter, all services can be initialized from the command-line by running the
services Bash script provided within the
repository:

./services <command>

Where <command> will vary depending upon the databases we wish to activate. This command will also import seed data
from the previous tutorials and provision the dummy IoT sensors on startup.

ℹ️Note: If you want to clean up and start over again you can do so with the following command:

./services stop

MongoDB - Persisting Context Data into a Database

Persisting historic context data using MongoDB technology is relatively simple to configure since we are already using a
MongoDB instance to hold data related to the Orion Context Broker and the IoT Agent. The MongoDB instance is listening
on the standard 27017 port and the overall architecture can be seen below:

MongoDB - Start up

To start the system with a MongoDB database only, run the following command:

./services mongodb

Checking the Cygnus Service Health

Once Cygnus is running, you can check the status by making an HTTP request to the exposed CYGNUS_API_PORT port. If the
response is blank, this is usually because Cygnus is not running or is listening on another port.

You should see several containers running. If cygnus is not running, you can restart the containers as necessary.

Generating Context Data

For the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated. The
dummy IoT Sensors can be used to do this. Open the device monitor page at http://localhost:3000/device/monitor and
unlock a Smart Door and switch on a Smart Lamp. This can be done by selecting an appropriate the command from
the drop down list and pressing the send button. The stream of measurements coming from the devices can then be seen
on the same page:

Subscribing to Context Changes

Once a dynamic context system is up and running, we need to inform Cygnus of changes in context.

This is done by making a POST request to the /v2/subscription endpoint of the Orion Context Broker.

The fiware-service and fiware-servicepath headers are used to filter the subscription to only listen to
measurements from the attached IoT Sensors, since they had been provisioned using these settings

The idPattern in the request body ensures that Cygnus will be informed of all context data changes.

The notification url must match the configured CYGNUS_API_PORT

The attrsFormat=legacy is required since Cygnus currently only accepts notifications in the older NGSI v1 format.

Within the notification section of the response, you can see several additional attributes which describe the health
of the subscription

If the criteria of the subscription have been met, timesSent should be greater than 0. A zero value would indicate
that the subject of the subscription is incorrect or the subscription has created with the wrong fiware-service-path
or fiware-service header

The lastNotification should be a recent timestamp - if this is not the case, then the devices are not regularly
sending data. Remember to unlock the Smart Door and switch on the Smart Lamp

The lastSuccess should match the lastNotification date - if this is not the case then Cygnus is not receiving
the subscription properly. Check that the hostname and port are correct.

Finally, check that the status of the subscription is active - an expired subscription will not fire.

MongoDB - Reading Data from a database

To read MongoDB data from the command-line, we will need access to the mongo tool run an interactive instance of the
mongo image as shown to obtain a command-line prompt:

docker run -it --network fiware_default --entrypoint /bin/bash mongo

You can then log into to the running mongo-db database by using the command-line as shown:

mongo --host mongo-db

Show Available Databases on the MongoDB server

To show the list of available databases, run the statement as shown:

Query:

show dbs

Result:

The result include two databases admin and local which are set up by default by MongoDB, along with four
databases created by the FIWARE platform. The Orion Context Broker has created two separate database instance for each
fiware-service

The Store entities were created without defining a fiware-service and therefore are held within the orion
database, whereas the IoT device entities were created using the openiotfiware-service header and are held
separately. The IoT Agent was initialized to hold the IoT sensor data in a separate MongoDB database called
iotagentul.

As a result of the subscription of Cygnus to Orion Context Broker, a new database has been created called sth_openiot.
The default value for a Mongo DB database holding historic context consists of the sth_ prefix followed by the
fiware-service header - therefore sth_openiot holds the historic context of the IoT devices.

Read Historical Context from the server

Query:

use sth_openiot
show collections

Result:

Looking within the sth_openiot you will see that a series of tables have been created. The names of each table consist
of the sth_ prefix followed by the fiware-servicepath header followed by the entity ID. Two table are created for
each entity - the .aggr table holds some aggregated data which will be accessed in a later tutorial. The raw data can
be seen in the tables without the .aggr suffix.

The historical data can be seen by looking at the data within each table, by default each row will contain the sampled
value of a single attribute.

The usual MongoDB query syntax can be used to filter appropriate fields and values. For example to read the rate at
which the Motion Sensor with the id=Motion:001_Motion is accumulating, you would make a query as follows:

To leave the MongoDB client and leave interactive mode, run the following:

exit

exit

PostgreSQL - Persisting Context Data into a Database

To persist historic context data into an alternative database such as PostgreSQL, we will need an additional
container which hosts the PostgreSQL server - the default Docker image for this data can be used. The PostgreSQL
instance is listening on the standard 5432 port and the overall architecture can be seen below:

We now have a system with two databases, since the MongoDB container is still required to hold data related to the Orion
Context Broker and the IoT Agent.

Port 5432 is the default port for a PostgreSQL server. It has been exposed so you can also run the pgAdmin4 tool
to display database data if you wish

The postgres-db container is driven by environment variables as shown:

Key

Value.

Description

POSTGRES_PASSWORD

password

Password for the PostgreSQL database user

POSTGRES_USER

postgres

Username for the PostgreSQL database user

POSTGRES_DB

postgres

The name of the PostgreSQL database

ℹ️Note: Passing the Username and Password in plain text environment variables like this is a
security risk. Whereas this is acceptable practice in a tutorial, for a production environment, you can avoid this
risk by applying Docker Secrets

ℹ️Note: Passing the Username and Password in plain text environment variables like this is a
security risk. Whereas this is acceptable practice in a tutorial, for a production environment,
CYGNUS_POSTGRESQL_USER and CYGNUS_POSTGRESQL_PASS should be injected using
Docker Secrets

PostgreSQL - Start up

To start the system with a PostgreSQL database run the following command:

./services postgres

Checking the Cygnus Service Health

Once Cygnus is running, you can check the status by making an HTTP request to the exposed CYGNUS_API_PORT port. If the
response is blank, this is usually because Cygnus is not running or is listening on another port.

You should see several containers running. If cygnus is not running, you can restart the containers as necessary.

Generating Context Data

For the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated. The
dummy IoT Sensors can be used to do this. Open the device monitor page at http://localhost:3000/device/monitor and
unlock a Smart Door and switch on a Smart Lamp. This can be done by selecting an appropriate the command from
the drop down list and pressing the send button. The stream of measurements coming from the devices can then be seen
on the same page:

Subscribing to Context Changes

Once a dynamic context system is up and running, we need to inform Cygnus of changes in context.

This is done by making a POST request to the /v2/subscription endpoint of the Orion Context Broker.

The fiware-service and fiware-servicepath headers are used to filter the subscription to only listen to
measurements from the attached IoT Sensors, since they had been provisioned using these settings

The idPattern in the request body ensures that Cygnus will be informed of all context data changes.

The notification url must match the configured CYGNUS_API_PORT

The attrsFormat=legacy is required since Cygnus currently only accepts notifications in the older NGSI v1 format.

As you can see, the database used to persist context data has no impact on the details of the subscription. It is the
same for each database. The response will be 201 - Created

PostgreSQL - Reading Data from a database

To read PostgreSQL data from the command-line, we will need access to the postgres client, to do this, run an
interactive instance of the postgresql-client image supplying the connection string as shown to obtain a command-line
prompt:

Query:

Result:

As a result of the subscription of Cygnus to Orion Context Broker, a new schema has been created called openiot. The
name of the schema matches the fiware-service header - therefore openiot holds the historic context of the IoT
devices.

Read Historical Context from the PostgreSQL server

Once running a docker container within the network, it is possible to obtain information about the running database.

The usual PostgreSQL query syntax can be used to filter appropriate fields and values. For example to read the rate
at which the Motion Sensor with the id=Motion:001_Motion is accumulating, you would make a query as follows:

To leave the Postgres client and leave interactive mode, run the following:

\q

You will then return to the command-line.

MySQL - Persisting Context Data into a Database

Similarly, to persisting historic context data into MySQL, we will again need an additional container which hosts
the MySQL server, once again the default Docker image for this data can be used. The MySQL instance is listening on the
standard 3306 port and the overall architecture can be seen below:

Once again we have a system with two databases, since the MongoDB container is still required to hold data related to
the Orion Context Broker and the IoT Agent.

MySQL - Database Server Configuration

ℹ️Note: Using the default root user and displaying the password in an environment variables
like this is a security risk. Whereas this is acceptable practice in a tutorial, for a production environment, you can
avoid this risk by setting up another user and applying
Docker Secrets

The mysql-db container is listening on a single port:

Port 3306 is the default port for a MySQL server. It has been exposed so you can also run other database tools to
display data if you wish

The mysql-db container is driven by environment variables as shown:

Key

Value.

Description

MYSQL_ROOT_PASSWORD

123.

specifies a password that is set for the MySQL root account.

MYSQL_ROOT_HOST

postgres

By default, MySQL creates the root'@'localhost account. This account can only be connected to from inside the container. Setting this environment variable allows root connections from other hosts

ℹ️Note: Passing the Username and Password in plain text environment variables like this is a
security risk. Whereas this is acceptable practice in a tutorial, for a production environment, CYGNUS_MYSQL_USER
and CYGNUS_MYSQL_PASS should be injected using
Docker Secrets

The cygnus container is listening on two ports:

The Subscription Port for Cygnus - 5050 is where the service will be listening for notifications from the Orion
context broker

The Management Port for Cygnus - 5080 is exposed purely for tutorial access - so that cUrl or Postman can make
provisioning commands without being part of the same network.

MySQL - Start up

To start the system with a MySQL database run the following command:

./services mysql

Checking the Cygnus Service Health

Once Cygnus is running, you can check the status by making an HTTP request to the exposed CYGNUS_API_PORT port. If the
response is blank, this is usually because Cygnus is not running or is listening on another port.

You should see several containers running. If cygnus is not running, you can restart the containers as necessary.

Generating Context Data

For the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated. The
dummy IoT Sensors can be used to do this. Open the device monitor page at http://localhost:3000/device/monitor and
unlock a Smart Door and switch on a Smart Lamp. This can be done by selecting an appropriate the command from
the drop down list and pressing the send button. The stream of measurements coming from the devices can then be seen
on the same page:

Subscribing to Context Changes

Once a dynamic context system is up and running, we need to inform Cygnus of changes in context.

This is done by making a POST request to the /v2/subscription endpoint of the Orion Context Broker.

The fiware-service and fiware-servicepath headers are used to filter the subscription to only listen to
measurements from the attached IoT Sensors, since they had been provisioned using these settings

The idPattern in the request body ensures that Cygnus will be informed of all context data changes.

The notification url must match the configured CYGNUS_API_PORT

The attrsFormat=legacy is required since Cygnus currently only accepts notifications in the older NGSI v1 format.

As you can see, the database used to persist context data has no impact on the details of the subscription. It is the
same for each database. The response will be 201 - Created

MySQL - Reading Data from a database

To read MySQL data from the command-line, we will need access to the mysql client, to do this, run an interactive
instance of the mysql image supplying the connection string as shown to obtain a command-line prompt:

Result:

As a result of the subscription of Cygnus to Orion Context Broker, a new schema has been created called openiot. The
name of the schema matches the fiware-service header - therefore openiot holds the historic context of the IoT
devices.

Read Historical Context from the MySQL server

Once running a docker container within the network, it is possible to obtain information about the running database.

The usual MySQL query syntax can be used to filter appropriate fields and values. For example to read the rate at
which the Motion Sensor with the id=Motion:001_Motion is accumulating, you would make a query as follows:

Hostname of the PostgreSQL server used to persist historical context data

CYGNUS_POSTGRESQL_PORT

5432

Port that the PostgreSQL server uses to listen to commands

CYGNUS_POSTGRESQL_USER

postgres

Username for the PostgreSQL database user

CYGNUS_POSTGRESQL_PASS

password

Password for the PostgreSQL database user

CYGNUS_MYSQL_HOST

mysql-db

Hostname of the MySQL server used to persist historical context data

CYGNUS_MYSQL_PORT

3306

Port that the MySQL server uses to listen to commands

CYGNUS_MYSQL_USER

root

Username for the MySQL database user

CYGNUS_MYSQL_PASS

123

Password for the MySQL database user

CYGNUS_LOG_LEVEL

DEBUG

The logging level for Cygnus

Multi-Agent - Start up

To start the system with multiple databases run the following command:

./services multiple

Checking the Cygnus Service Health

Once Cygnus is running, you can check the status by making an HTTP request to the exposed CYGNUS_API_PORT port. If the
response is blank, this is usually because Cygnus is not running or is listening on another port.

You should see several containers running. If cygnus is not running, you can restart the containers as necessary.

Generating Context Data

For the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated. The
dummy IoT Sensors can be used to do this. Open the device monitor page at http://localhost:3000/device/monitor and
unlock a Smart Door and switch on a Smart Lamp. This can be done by selecting an appropriate the command from
the drop down list and pressing the send button. The stream of measurements coming from the devices can then be seen
on the same page:

Subscribing to Context Changes

Once a dynamic context system is up and running, we need to inform Cygnus of changes in context.

This is done by making a POST request to the /v2/subscription endpoint of the Orion Context Broker.

The fiware-service and fiware-servicepath headers are used to filter the subscription to only listen to
measurements from the attached IoT Sensors

The idPattern in the request body ensures that Cygnus will be informed of all context data changes.

The attrsFormat=legacy is required since Cygnus currently only accepts notifications in the older NGSI v1 format.

The throttling value defines the rate that changes are sampled.

When running in multi-agent mode, the notification url for each subscription must match the defaults for the given
database.

The default port mapping can be seen below:

sink

port

mysql

5050

mongo

5051

ckan

5052

hdfs

5053

postgresql

5054

cartodb

5055

Since this subscription is using port 5050 the context data will eventually be persisted to the MySQL database.