Esper: High Volume Event Stream Processing and Correlation in Java

Posted by Floyd Marinescu on Jul 28, 2006

Sections

Architecture & Design,

Development,

Enterprise Architecture

Topics

Workflow / BPM ,

Java ,

Messaging

Tags

Esper ,

Event Stream Processing

Esper is an event stream processing (ESP) and event correlation engine (CEP) unveiled this week with a 1.0 launch on Codehaus.   Esper is a real time engine that triggers actions when event conditions occurs among event streams. It is designed for high volume event correlation where millions of events coming in would make it impossible to store them all to later query them using a classical database architecture.

Esper acts as a message consumer in a Java application, consuming immutable plain java beans sent by the application and acting upon them when matching conditions have occured.  To evaluate the java beans, Esper compares against developer-written Event Query Language (EQL) statements which are matched against events as they come in and used to trigger POJO listener/handler classes.  EQL allows the expressing of complex matching conditions that include temporal windows, and joining different event streams, filtering, sorting, grouping, etc:

Esper was founded by Thomas Bernhardt and emerged out of protypes built by Thomas for a large financial institution that had asked him to evaluate different rules engines for use in an application for monitoring a trading system. The rules engines proved cumbersome and slow for this usecase, what was really needed was real time event correlation engine.  InfoQ spoke to Alexandre Vasseur who is helping Thomas grow the Esper community.  Many may remember Alexandre as one of three committers on the AspectWerkz project (which has since merged with AspectJ), who later joined BEA. 

Alexandre provided an example of computing the average stock price of the BEA tick on a sliding window of 30 seconds. Given a StockTick event bean with a price and symbol property and the EQL "select avg(price) from StockTick.win:time(30) where symbol='BEA'", a POJO would get notified as tick come in.  Statements can be much more complex, and also be combined together with "followed by" conditions.

On the differences between Esper and messaging or rules engines: 

Today, event based system are often confused with message based system such as JMS and alike backends or Enterprise Service Bus (ESB). Messaging system are event based but usually do not correlate events or process event streams at all. They simply process unitary events - and it becomes quickly complex if you need to correlate messages... [Esper] is not to be confused with classical rules engines...The internals of Esper are made up of fairly complex algorithms primarily relying on state machines and delta networks in which only changes to data are communicated across object boundaries when required.

On the best uses and mis-uses for Esper, Alexandre replied: 

Esper is best suited for realtime event driven applications. Typical application areas are business process management and automation, finance, network and application monitoring and sensor network applications. Esper take much of the complexity out of developing applications that detect patterns among events, filter events, aggregate time or length windows of events, join event streams, trigger based on absence of events etc.

Esper is not designed for storing and retrieval of fairly static data - that is better left to conventional databases. In-memory databases may be better suited to CEP applications than traditional relational databases as they generally have good query performance. Yet they are not optimized to provide immediate, real-time query results required for CEP and event stream analysis.

According to Alexandre,  Esper is the only open source Event Stream Processing solution; but the initial version lacks high availability features in commercial ESP engines such as StreamBase or Coral8. Esper can run standalone, the 1.0 release assumes events to be POJOs. The Esper team is currently working on providing native XML format support.

1. Back to top

   Sounds interesting for a variety of uses cases - RSS, security detections?

   by Floyd Marinescu

   This sounds pretty interesting and I can see an event processing query language amd framework like this being useful in a variety of use cases beyond financial or high volume. Imagine using it to consume thousands of RSS feeds and scan for patterns and do useful things with that info, or perhaps scanning incoming web requests in memory and detecting denial of service attacks or simply upsurges in requests/minute and being able to respond to that in smart ways.
   
   Very cool!

   Reply

2. Back to top

   Market Surveillance

   by Horia Muntean

   Yummy!
   And real-time risk management.
   
   Regards,
   Horia

   Reply

3. Back to top

   ILOG JRules can also do this

   by Daniel Selman

   I will have to download Esper and take a look, but ILOG Rules (C++) and JRules have been used for realtime event processing (both in telco and finance) for a number of years. For example you can write IRL code such as that included below (for event correlation and filtering). The RetePlus algorithm in JRules is an optimized pattern-matching mechanism that supports temporal reasoning. It is also possible to use the JRules tools to built domain-specific languages for business users that generate the code included below. JRules can reason on unmodified POJOs as well as XML data.
   
   

   
   This rule detects that two alarms with the same Communications probable cause have occurred 
   on two pieces of equipment which are connected together, and synthesizes a new alarm on the 
   connecting link. In this rule the handling of partial order is illustrated. The 
   id > alarm1.id test is here to break the symmetry between e1 and e2, and thus avoid the
   rule being fired twice.
   
   when {
   	alarm1 : event Alarm(e1 : equipment;
   	   perceivedSeverity != Cleared;
   	   probableCause == Communications);
   	alarm2 : event Alarm(e2 : equipment;
   	   perceivedSeverity != Cleared;
   	   probableCause == Communications;
   	   e2 != e1;
   	   id > alarm1.id;
   	   this before[-10, 10] alarm1);
   	link : EquipmentLink(isConnectedTo(e1);
   	isConnectedTo(e2));
   }
   then 
   {
          insert event Alarm(link, link.newCorrelatedAlarmId(), Equipment, Major);
   }
   

   
   
   Or this rule, which does event filtering:
   
   

   
   This rule suppresses an alarm and its clearing alarm if the clearing alarm occurs within 
   two seconds after the initial alarm.
   
   when {
     alarm1 : event Alarm(perceivedSeverity != Alarm.Cleared);
     alarm2 : event Alarm(equipment == alarm1.equipment;
       id == alarm1.id;
       perceivedSeverity == Alarm.Cleared;
   	this after[0, 2] alarm1);
   }
   then 
   {
   	retract alarm1;
   	retract alarm2;
   }
   

   
   
   Thanks for the link to Esper!
   
   Sincerely,
   
   Daniel Selman
   (ILOG)

   Reply

4. Back to top

   Re: ILOG JRules can also do this

   by Alex Vasseur

   Hi Daniel
   Thanks for your comment. I believe ILOG JRules and more generally Drools or Jess or any Rete based production rule engine can indeed be used to address part of what CEP coins. Especially triggering by correlating events, possibly including a temporal relationship between events as your sample shows.
   I think ISphere folks have written about "delta dataflow" algorithms beeing a generalization of Rete ones.
   
   That said I believe the ESP side - Event Stream Processing - is a different beast, where what matters is the "S" for Stream. In this side of the Esper engine we provide several language facilities to build expressions using time not for temporal relationship (happened before etc) but for sliding window.
   
   As an example, this makes it very valuable for computing things like volume weigthed average (VWAP) of ticks - which would be (I believe, correct me if I am wrong) awfull using something Rete based like JRules. In Esper I'd wrote it as follow:
   

   
   select * from StockTickEvent(symbol='BEA').win:time(3000).stat:weighted_avg('price', 'volume')
   

   
   
   I believe beeing able to combine both CEP and ESP is a key value add of Esper. You can read more about how the two play together in our tutorials.
   
   Alex

   Reply

5. Back to top

   Re: ILOG JRules can also do this

   by Mark Proctor

   I've been thinking on CEP and ESP, both are definitely possible in Rete - they will be a strong part of the work I'll do early next year. I have splashed some initial thoughts on ESP on the Drools wiki.
   
   wiki.jboss.org/wiki/Wiki.jsp?page=EventStreamAn...
   
   As Drools allows for querries we can easily provde ECA type syntax. Maybe we should be looking to join the projects together and build ESP/ECA on the drools platform?

   Reply

6. Back to top

   Re: ILOG JRules can also do this

   by Jin Chun

   Not to sound stupid, but I am literally drooling at the prospect of drools integration and Esper in general. This is the sort of stuff that is being pushed quite heavily right now, and having a first class OS alternative would be more than fantastic.

   Reply

7. Back to top

   Re: ILOG JRules

   by Gaurav Shrinivas

   Can any one help me with a sample "Suppress rule" in iLog JRules.
   
   Please
   
   Regards
   Gaurav
   
   gauravshrinivas at yahoo dot com

   Reply

8. Back to top

   Re: ILOG JRules can also do this

   by Gaurav Shrinivas

   Daniel
   Please help with a code to suppress Alarms in iLog JRules.
   
   my mail id is given.

   Reply

9. Back to top

   Re: ILOG JRules can also do this

   by Gaurav Shrinivas

   Friends,
   
   Thanks as I was able to finish it well.

   Reply

• Older >