Complex Event Processing Made Easy (using Esper)

Note - a full working example is available over on GitHub:

What is Complex Event processing (CEP)?

Complex Event Processing (CEP), or Event Stream Stream Processing (ESP) are technologies commonly used in Event-Driven systems. These type of systems consume, and react to a stream of event data in real time. Typically these will be things like financial trading, fraud identification and process monitoring systems – where you need to identify, make sense of, and react quickly to emerging patterns in a stream of data events.

Key Components of a CEP system

A CEP system is like your typical database model turned upside down. Whereas a typical database stores data, and runs queries against the data, a CEP data stores queries, and runs data through the queries.

To do this it basically needs:

Data – in the form of ‘Events’

Queries – using EPL (‘Event Processing Language’)

Listeners – code that ‘does something’ if the queries return results

A Simple Example - A Nuclear Power Plant

Take the example of a Nuclear Power Station..

Now, this is just an example – so please try and suspend your disbelief if you know something about Nuclear Cores, Critical Temperatures, and the like. It’s just an example. I could have picked equally unbelievable financial transaction data. But ...

Monitoring the Core Temperature

Now I don’t know what the core is, or if it even exists in reality – but for this example lets assume our power station has one, and if it gets too hot – well, very bad things happen..

Lets also assume that we have temperature gauges (thermometers?) in place which take a reading of the core temperature every second – and send the data to a central monitoring system.

What are the requirements?

We need to be warned when 3 types of events are detected:

MONITOR

just tell us the average temperature every 10 seconds - for information purposes

WARNING

WARN us if we have 2 consecutive temperatures above a certain threshold

CRITICAL

ALERT us if we have 4 consecutive events, with the first one above a certain threshold, and each subsequent one greater than the last – and the last one being 1.5 times greater than the first. This is trying to alert us that we have a sudden, rising escalating temperature spike – a bit like the diagram below. And let’s assume this is a very bad thing.

Using Esper

There are a number of ways you could approach building a system to handle these requirements. For the purpose of this post though - we will look at using Esper to tackle this problem

How we approach this with Esper is:

Using Esper – we can create 3 queries (using EPL - Esper Query Language) to model each of these event patterns.

We then attach a listener to each query - this will be triggered when the EPL detects a matching pattern of events)

We create an Esper service, and register these queries (and their listeners)

We can then just throw Temperature data through the service – and let Esper tell alert the listeners when we get matches.

(A working example of this simple solution is available on Githib - see link above)

Some Code Snippets

TemperatureEvent

We assume our incoming data arrives in the form of a TemperatureEvent POJO

If it doesn't - we can convert it to one, e.g. if it comes in via a JMS queue, our queue listener can convert it to a POJO. We don't have to do this, but doing so decouples us from the incoming data structure, and gives us more flexibility if we start to do more processing in our Java code outside the core Esper queries. An example of our POJO is below

The Running Demo

An example of the running demo is shown below - it generates random Temperature events and sends them through the Esper processor (in the real world this would come in via a JMS queue, http endpoint or socket listener).

When any of our 3 queries detect a match - debug is dumped to the console. In a real world solution each of these 3 listeners would handle the events differently - maybe by sending messages to alert queues/endpoints for other parts of the system to pick up the processing.

Conclusions

Using a system like Esper is a neat way to monitor and spot patterns in data in real time with minimal code.

This is obviously (and intentionally) a very bare bones demo, barely touching the surface of the capabilities available. Check out the Esper web site for more info and demos.

Esper also has a plugin for Apache Camel Integration engine - which allows you to configure you EPL queries directly in XML Spring camel routes, removing the need for any Java code completely (we will possibly cover this in a later blog post!)

There are thousands of ways to make money, however just one can give you that great financial breakthrough. Every online opportunity must be carefully investigated before starting. Here is the great opportunity to secure your investment and earn more . email processing system