Perspective on the Business of Event Processing

April 2011

April 21, 2011

The philosophy of Erlang and StreamBase align quite well. Quoting Mike Williams, one of the three inventors of Erlang:

Find the right methods—Design by Prototyping

It is not good enough to have ideas, you must also be able to implement them and know they work

Make mistakes on a small scale, not in a production project

These philosophies of rapid development also apply to the way we've designed StreamBase; additionally, our customers also demand that much less technical users participate in the rapid implementation of ideas. StreamBase has a visual development approach that allows less technical, more "business" users - with mathematics, analytics, and business domain expertise - use the same tools as professional software developers.

What if we could add Erlang developers to that party? Event-driven, visual development, coupled with the power and rapid development of Erlang?

Well you can. Darach Ennis (@darachennis) shows us how in his recent presentation at Erlang UG in London.

StreamBase has a graphical, compiler-oriented programming language, targeted at serious developers, that can also be used by the less technical. Erlang programmers, and the Erjang folks with whom we share a virtual machine, do their thing in Erlang, and that code becomes a graphical "operator" on the StreamBase Studio pallet. In Darach's example he's used Erlang to implement a MACD analytic, and integrate that into a StreamBase-based trading system so it operates in the event-driven CEP application.

The combination of StreamBase and Erlang combines the best of both worlds: StreamBase for rapid, graphical development to develop low latency applications that process 100's of thousands of events a second. Combining "how," "why," and "under what conditions," real time streams are matched and processed. StreamBase allows business analysts, actuaries, quantitative contributors and developers to share a common language. An event-driven integration whiteboard, if you will, that compiles to efficient optimized native binary code. Add Erlang for development of analytics like MACD and you have a powerful combination.

Erlang support will be contributed soon to the StreamBase Component Exchange (SBX), our public "app exchange" for CEP integrations, applications, frameworks, and sample code. You'll already find support for Java, C++, R, Matlab, python, .NET. If you want to plug in some cool analytics written in C++, just do it. You want to exploit Basho's Riak? Sure, plug it in.

As a result, StreamBase allows developers control over how they exploit the facilities at their disposal. Use our graphical DSL for flow oriented algorithms and real time low latency high frequency data management. It's high performance from a human productivity and systemic perspective.

Learn more by watching Darach's presentation here, and follow him on Twitter at @darachennis.

April 12, 2011

Today we announced SteamBase 7, the next major version of our market-leading Complex Event Processing ￼(CEP) software. In press and analyst pre-briefings last week, we discussed a lot of cool new features. Here’s a quick tour.

A HUGE STEP FORWARD IN CEP USABILITY

Chief among the innovations of StreamBase 7 is the revamped StreamBase Studio. The interface innovations of the iPhone, iPad, and modern operating systems has made great interface design expected. StreamBase has always been the best and easiest to use CEP platform. But with StreamBase 7, our engineering team has taken a huge leap forward in usability.

To achieve this usability, StreamBase 7 has new code organizational features, integrated annotation features for English descriptions of functions, gorgeous diagram rendering techniques, and facilities to isolate and encapsulate the necessary complexity of a sophisticated application. Literally, StreamBase users can now read StreamBase application code at a glance, and organize it effectively so that the Business and IT can collaborate in a way never possible before with any CEP product. These features that make this all happen include:

A beautifully redesigned user experience, including the use of transparency (similar to Windows 7 Aero), overlays to communicate information at a glance (e.g., iOS number-in-red-circle), gradients rather than solid color to ease readability (think in terms of Windows 7 and iOS) and, and complete color customization (a la Windows 7 themes.)

Support for custom modules (groups of application logic) and extension point styles. Better control over modularity allows IT to manage large applications, and manage how and where business users or other groups can contribute code. For example, a risk domain expert can create risk checks in one module while IT focusses on connectivity, performance and integration in other modules.

Advanced graphical code layout, with new group layout features and code rendering controls

A new rendering look and feel to ensure code is easier to follow at a glance

New mouse and keyboard gestures to increase EventFlow coding productivity, such as on-canvas renaming of components

New hotkeys that give the ability to instantly create CEP language operators

Layout-alignment shortcuts

The result is that business users and IT can work in one environment, with one set of tools, with one set of code, dramatically improving time to market, quality and ability to evolve systems once they are deployed.

In recent years, StreamBase deployments have become bigger, more global, and more complicated. So StreamBase 7 improves usability throughout the entire software development lifecycle, including:

New code refactoring schemes. So developers can quickly identify changes in their application logic and adjust the rest of the application more quickly as modifications are made.

400% faster application execution while debugging. So problems can be identified more quickly

The ability to attach Studio to a running StreamBase server. For better diagnostics in deployment.

A new XML editor. Allows operations to more effectively configure StreamBase deployments.

Dynamic extension points that allow users to configure StreamBase applications to be configured quickly at runtime. When a new algorithm or analytic is produced by a developer or business users, this code can be dynamically inserted into the StreamBase code, tested, and deployed.

Support for 64-bit workstations for very large StreamBase application development

PERFORMANCE IMPROVEMENTS IN FOUR DIMENSIONS

From the first release of StreamBase, the focus has been on performance. Previous SteamBase performance innovations include the StreamSQL compiler, the ability to visualize parallelism and scalability, data parallelism and dispatch, and our open architecture for highly available systems all combine to create the fastest, most scalable platform for ultra low latency applications.

Version 7 of StreamBase continues to break down CEP performance barriers in four dimensions: ultra low latency connectivity, predictability of latency, FIX performance, and with high-speed computing technologies.

1: MICROSECOND-LEVEL LATENCY

All CEP platforms have an architecture for absorbing and emitting events. Some run out of the main CEP engine process, which can add significant latency overhead to event processing. StreamBase already features an super-fast in-process adapter architecture, but goes further in StreamBase 7 with an optimized “zero-context-switch adapter architecture” which allows StreamBase applications to process messages in a single thread from the time it enters the engine, through StreamBase logic, to the time it leaves the engine. This architecture allows messages to pass through the StreamBase engine in microseconds, and has been optimized to speed up CEP application performance.

2: PREDICTABILITY OF LATENCY

The predictability of performance has been improved for Streambase 7 applications, as users now have access to server configuration options that provide the user with fine-grain control of the tradeoffs between ultra-low-latency and throughput. And, overall platform improvements have yielded over 20% improvement in application performance across the board.

3: ULTRA LOW LATENCY FIX CONNECTIVITY

Many StreamBase customers in the capital markets is already known for its low latency connectivity architecture and version 7 comes with over 22 new connectivity options. Many of those connectivity options leverage FIX, and in StreamBase 7 FIX connectivity has been dramatically enhanced. In benchmark testing, the new StreamBase FIX engine has clocked in at 50-75 microseconds send time depending on message size, all while running on Linux and commodity hardware.

The state of the art of ultra-low-latency is rapidly expanding, and many StreamBase ￼customers have been on the leading edge of applying new technologies to deal with big data in innovative ways.

GPUs are being used to continuously produce metrics, factors, and parametrics, contribute them to memory data structures for ultra low latency streams of data to adapt in flight to compute accelerated algorithms. They are also used to easily integrate accelerated compute and messaging or to perform collaborate distributed event processing in a high level flow oriented language.

To support this trend for high-speed computing with StreamBase, we have added several connectivity options in StreamBase 7 to make it easier for developers to connect StreamBase to their existing and next generation IT infrastructure, including hardware-based middleware for ultra-low latency operations, including enhanced low latency middleware connectivity such as Wombat, 29West, Solace, and Tervela.

REACTION TO STREAMBASE 7

Analyst Adam Honore’ said at the end of our briefing: “wow, you’ve been busy!” and he’s right - as customers build more sophisticated applications with StreamBase, with bigger demands from the business users, and bigger demands from IT, with the ever-increasing need for speed, StreamBase 7 extends our lead as the best-in-class CEP platform in the industry.

The benefit of bringing the business and IT together is faster time to market, with higher quality. With tight communication, the business and IT make mistakes faster, learn from them faster, and innovate faster. As Thomas Edison said: “I make more mistakes than anyone I know. And sooner or later, I patent them.”

But my favorite anecdote comes from one of our customers - Chris Lees, vice president of SunGard’s global trading business. He put it this way: "The usability of StreamBase Studio helped us develop a new Valdi product, we used it to describe requirements collaboratively with IT and because we were working with the actual code, we made better architectural decisions early that made for a better first release.”