Demystifying IBM Streams

The aim of this blog is to help you build a picture of how one could develop SPL programs in IBM streams. In this blog, you will learn about —

Data Streaming and the need for real-time decision systems.

IBM Streams and its toolkits.

Some SPL code.

Let us take a deep dive into the ocean of IBM Streams.

50 m Atlantis 1 Submarine — Streaming Analytics

Streaming Analytics, (also referred to as Stream Processing) is the method of processing huge data (Stream) in-flight. These data streams are the uninterrupted flow of an overlong sequence of data. The power of streaming analytics is such that it permits for the streaming of millions of events per sec and entitles us to build applications that recommend real-time decision making.

By the year 2021, the global IP traffic will reach 3.3 ZettaBytes (ZB) per year – Cisco

As hardware gets cheaper and devices get smarter the volume of data being sent and receive is dramatically growing. Faced with this growing volume of constantly changing data, today’s firms are challenged to make informed, real-time business decisions and stay ahead of the competition without the need to cache the data in Hadoop Systems.

300m Scuba Divers — What is IBM Streams?

IBM streams (IBM Infosphere Streams) is a proprietary tool which has a development platform as well as runtime environment that enables establishments to develop and execute applications that ingest, filter and analyze enormous volumes of data streams.

It uses its unique Streams Processing Language (SPL) and can be extended with C, C++ or Java applications to reuse existing logic. It consists of an Eclipse-based IDE, application graphs and runtime monitoring to simplify the process of building and managing applications.

Here, it uses a Beacon operator to generate tuples that carry “Hello World” messages. A custom sink operator receives the tuples from Beacon and displays it 5 times on the console.

Few takeaways —

Stream

A stream is a sequence of tuples and tuples are collections of attributes. (Ex. rstring message). A stream can be an infinite sequence of tuples, such as a stream from a network sensor.

Alternatively, a stream can be finite, such as a stream that is created from the contents of a file.

Tuple An individual piece of data in a stream. These data can be structured or unstructured.

Operator

These are the basic building blocks of stream processing application. They process tuples from an incoming stream and produces an output stream.

Operators process an input stream in one of the following logical modes.

As each tuple is received.

As a window.

This also manipulates the tuple data from the input stream and manufactures the results in the form of an output stream.

On successful compilation of this spl code, a .sab file is produced. A streams application bundle (sab) file is a single, relocatable file that contains all the artifacts that are needed to run your application. When this is submitted for execution, the application bundle file is then unbundled into a runtime application.

1000m Giant Pacific Octopus — Counting lines inside all Files of a Specified Directory using a pattern to match the file types.

This operator parses individual DNS message field received in input tuples and emits tuples containing DNS message data. Users can then use DNS parser result functions to further process the payload data.

7700m Deepest Water Fishes — Custom Operator

The Custom operator is a special logic-related operator that can receive and send any number of streams and does not do anything by itself. Thus, it offers a blank slate for customization. Let us see how we sort maps based on its values.

shutdownPE() — Shutdown processing element (PE). This Stops the current PE. Whats PE? When you compile a stream processing application, the operator and streams relationships that make up the data flow graph are broken down into a set of individual execution units called PEs.

This is not the end of our deep-dive. This ocean offers a lot more. The journey to the deepest point underwater is still on. I’ll hopefully discover some lost pirate treasure before my next post but you Keep Streaming!