Stream and store retail order data for analysis

Use Akka to implement a WebSockets endpoint and stream data to IBM Db2 Event Store

Summary

In this code pattern, we’ll build a Scala app that uses Akka to implement a WebSockets endpoint which streams data to a Db2 Event Store database. For our data, we’ll use online retail order details in CSV format. We’ll use Jupyter notebooks with Scala and Brunel to visualize the Event Store data.

Description

In order to quickly react to changes in your business, you need to have high-speed data collection for current event-driven and also historical analysis. We’re using thin, fast WebSockets clients to send messages via WebSockets and letting the Akka toolkit handle the message streams, transform the data, and feed it into Event Store.

Akka HTTP gives us a very easy way to provide the WebSockets endpoint. Akka Streams is a powerful and elegant way to move the data from source to sink. We’re using Akka Streams with Alpakka to parse and transform the data from CSV text strings or CSV files into Spark SQL Rows. We use a simple divertTo function to decide which messages are new orders and which ones are cancellations.

We’re providing code to implement EventStoreSink and EventStoreFlow. This ultimately pushes the transformed data into Event Store tables. Once the data is in Event Store, we use the bundled Jupyter notebooks environment to analyze and visualize the data.

This is a simple example you can run on a laptop, but it’s built with tools that are designed to scale. It can easily be extended, whether you want to build a client, use more Akka functionality to handle the data streams, or flex your data science muscles with Event Store and Jupyter notebooks.

After completing this code pattern, you’ll understand how to:

Implement WebSockets with Scala and Akka.

Use Alpakka Flows for CSV parsing and inserts into Event Store

Use Jupyter notebooks and Scala to interact with Event Store

Use Spark SQL and Brunel visualizations to analyze the data

Flow

Setup the database with a Jupyter Notebook.

Submit CSV data via WebSockets.

Use Akka and Alpakka to transform the data and feed it into Event Store.

Present the data with Brunel visualizations in a Jupyter Notebook.

Instructions

Get the detailed instructions in the README file. These steps will show you how to:

Akka HTTP WebSockets support Akka HTTP provides a stream-based implementation of the WebSocket protocol that hides the low-level details of the underlying binary framing wire-protocol and provides a simple API to implement services using WebSocket.

Akka Alpakka The Alpakka project is an open source initiative to implement stream-aware and reactive integration pipelines for Java and Scala.

Ingest and Analyze Streaming Event Data at Scale with IBM Db2 EventStore In the fast race of digital transformation, enterprises are being challenged to manage and analyze the exploding volumes of event data. To harness event data for timely actions, enterprises must develop and deploy increasing number of event driven applications. These new types of eventdriven, modern workloads impose unique set of challenges for analytic systems.

IBM Db2 Event Store Solution Brief Discover the features that make Db2 Event Store uniquely suited to drive faster, deeper insights and learn about its use cases and editions.