Community Tutorials

Google Cloud Platform Community tutorials submitted from the community do not
represent official Google Cloud Platform product documentation.

Application Background

The setup described in this tutorial addresses following scenario: At industrial facilities, sensors are installed to monitor the equipment on site. Sensor data is continuously streamed to the cloud. There it is handled by different components for various purposes, such as real-time monitoring and alerts, long-term data storage for analysis, performance improvement, and model training.

In both cases, sample temperature data is collected that is generated from
simulated devices. This data is transformed into other data formats, and is passed to
another GCP product for further data processing and analysis. Cloud Functions is
suitable for simple Extract/Transform/Load (ETL) processing, while Cloud
Dataflow can handle more sophisticated data pipelines that involve multiple
transformations, joins, windowing, and so on.

Cloud IoT Core can not only receive data from MQTT clients, but also can send
configuration data to clients. It can be used to control behavior of devices or
the surrounding environment.

Data structure

The sample MQTT client simulates devices and generates sample data with the
following attributes:

DeviceId: A unique identifier for individual devices.

Timestamp: A timestamp for when a temperature is measured.

Temperature: The measured temperature from the device.

Coordinates: The longitude and latitude of the device.

Architecture

The sample MQTT client simulates a device and sends sample data to Cloud IoT
Core, which transforms and redirects requests to a Cloud Pub/Sub topic. After
the data is stored in Cloud Pub/Sub, it is retrieved by two subscribers: a function
in Cloud Functions and a streaming job running in Cloud Dataflow.

This tutorial shows how data is transformed and processed in Cloud Functions and
Cloud Dataflow.

Objectives

This tutorial demonstrates how to:

Deploy a function to Cloud Functions that transforms temperature data into
JSON format and logs it to Stackdriver Logging.

Deploy a streaming application to Cloud Dataflow that transforms temperature
data into BigQuery row format and inserts it into BigQuery.

Run an MQTT client that generates simulated temperature and coordinates, and
then submits the data to Cloud IoT Core.

Create a Cloud Storage bucket

Set environment variables

To make it easier to run commands, you can set environment variables so that you
don't have to supply options for some values that you’ll use repeatedly. You
will create the corresponding resources in later steps.

Confirm that the device entities have been created with the corresponding threshold temperature value:

Deploy a Cloud Function

In this section, you set up a function that logs data that is sent to Cloud IoT
Core and is retrieved through Cloud Pub/Sub. It also compares the temperature
received against the threshold value in Cloud Datastore. If the threshold is
exceeded, an error is logged.