Monday, November 26, 2007

Pigeons, Complex Event Processing and how to make millions with JBoss Drools

Posted by
paul browne

I'm still waiting for the JBoss bouncers to hand me my coat and ask me to leave this blog. Mark gets to talk about Unifying Rules and Processes. Fernando and Michael are very proud of the 2nd version of the Business Rules WebApp . And I get to talk about pigeons. Yep, Pigeons; birds that fly, sometimes useful for carrying messages and have one hidden talent.

Thankfully the pigeons didn't go to work at a Bank in the City (have you ever seen anything with feathers drive a Ferrari?) . While the pigeons would be very good at responding to simple market events events (Market up , sell; Market Down , Buy). more complex analysis escapes them; For example ; if the market is down for the 30 mins, and Shares in Acme corp are down more than 10% than the average ,and I have seen 3 buy orders for that share in the last 60 seconds = I think the market is about to turn = buy shares in Acme corp.

Never mind pigeons; most humans would find that difficult - think about trying to read the stock ticker prices (the ones you see rolling across the screen at MSNBC) for all stocks, while trying to hold the buy and sell information for the last 30 minutes in your head. And do that not only for one , but for the couple of hundred different types of shares in the market. And while keeping an eye on your own trading position so that you're not exposed to one sector of the market (e.g. keeping enough cash , not too many property or technology shares. No wonder most traders make their millions and burn out before they're 30 - that sort of Complex Event Processing (CEP) will wear you out.

Most IT applications are like pigeons; they can only handle simple events. Press Button. Do something.

The way to make millions is to design applications that can handle these complex events, and apply sophisticated business rules to the (evolving) situation. And do it quickly enough (milliseconds) to seize the opportunity before somebody else does. A keep on doing it as long as the market is open.

I spent a little time studying the query compilation in Esper and it looks rather simple. If I understand the code correctly, it doesn't generate efficient query plans. the other limitation I see with esper is that performance will drop significantly if the rule count increases.

The temporal extensions I've blogged about can handle a broader range of temporal logic. I've already implemented a simple version of temporal logic in jamocha. The code is up on sourceforge. Hopefully edson can use that a point of reference and implement the same thing for drools. I also described an aggregate pattern extension for RETE, which is much more powerful than streamSql. It's similar to the concept of cubes in MOLAP.

There are a number of limitations with Esper and other low end systems, the most obvious of which is the limitation on the clock representation. Clock representations are to do with representing the time an event happened, was it when it entered the network, is there a threshold, is it progammatic. We are designing our system to support a number of clock representations. We are also looking into interval based event representations; which are bit like composite events that are made up of a number of smaller interval events (i.e. start, stop). Over all we are aiming at doing more complex temporal reasoning, so our scope is much bigger than that of streaming sql.

We'll be unveiling our vision on this with a usable prototype in January. We won't have everything done by then, but we'll have an initial base implementation and a roadmap.

Bitcoin HYIP - Invested $300 and Paid $900 in one hourGiant-investment is a private investment group Test Plan * 300% after one hourStandard Plan * 500% after one hourPremium Plan * 800% after one hourInvest Nowhttp://www.giant-investment.comPrincipal Guaranteedhttp://www.payinghyiponline.com/Giant%20investment.html