InfoQ Homepage News Esper: High Volume Event Stream Processing and Correlation in Java

# Esper: High Volume Event Stream Processing and Correlation in Java

Esper is an event stream processing (ESP) and event correlation engine (CEP) unveiled this week with a 1.0 launch on Codehaus.   Esper is a real time engine that triggers actions when event conditions occurs among event streams. It is designed for high volume event correlation where millions of events coming in would make it impossible to store them all to later query them using a classical database architecture.

Esper acts as a message consumer in a Java application, consuming immutable plain java beans sent by the application and acting upon them when matching conditions have occured.  To evaluate the java beans, Esper compares against developer-written Event Query Language (EQL) statements which are matched against events as they come in and used to trigger POJO listener/handler classes.  EQL allows the expressing of complex matching conditions that include temporal windows, and joining different event streams, filtering, sorting, grouping, etc:

Esper was founded by Thomas Bernhardt and emerged out of protypes built by Thomas for a large financial institution that had asked him to evaluate different rules engines for use in an application for monitoring a trading system. The rules engines proved cumbersome and slow for this usecase, what was really needed was real time event correlation engine.  InfoQ spoke to Alexandre Vasseur who is helping Thomas grow the Esper community.  Many may remember Alexandre as one of three committers on the AspectWerkz project (which has since merged with AspectJ), who later joined BEA.

Alexandre provided an example of computing the average stock price of the BEA tick on a sliding window of 30 seconds. Given a StockTick event bean with a price and symbol property and the EQL "select avg(price) from StockTick.win:time(30) where symbol='BEA'", a POJO would get notified as tick come in.  Statements can be much more complex, and also be combined together with "followed by" conditions.

On the differences between Esper and messaging or rules engines:
Today, event based system are often confused with message based system such as JMS and alike backends or Enterprise Service Bus (ESB). Messaging system are event based but usually do not correlate events or process event streams at all. They simply process unitary events - and it becomes quickly complex if you need to correlate messages... [Esper] is not to be confused with classical rules engines...The internals of Esper are made up of fairly complex algorithms primarily relying on state machines and delta networks in which only changes to data are communicated across object boundaries when required.
On the best uses and mis-uses for Esper, Alexandre replied:
Esper is best suited for realtime event driven applications. Typical application areas are business process management and automation, finance, network and application monitoring and sensor network applications. Esper take much of the complexity out of developing applications that detect patterns among events, filter events, aggregate time or length windows of events, join event streams, trigger based on absence of events etc.

Esper is not designed for storing and retrieval of fairly static data - that is better left to conventional databases. In-memory databases may be better suited to CEP applications than traditional relational databases as they generally have good query performance. Yet they are not optimized to provide immediate, real-time query results required for CEP and event stream analysis.
According to Alexandre,  Esper is the only open source Event Stream Processing solution; but the initial version lacks high availability features in commercial ESP engines such as StreamBase or Coral8. Esper can run standalone, the 1.0 release assumes events to be POJOs. The Esper team is currently working on providing native XML format support.

Style

## Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

• ##### Sounds interesting for a variety of uses cases - RSS, security detections?

by Floyd Marinescu /

• ##### Market Surveillance

by Horia Muntean /

• ##### ILOG JRules can also do this

by Daniel Selman /

• ##### Re: ILOG JRules can also do this

by Alex Vasseur /

• ##### Re: ILOG JRules can also do this

by Mark Proctor /

• ##### Re: ILOG JRules can also do this

by Jin Chun /

• ##### Re: ILOG JRules

by Gaurav Shrinivas /

• ##### Re: ILOG JRules can also do this

by Gaurav Shrinivas /

• ##### Re: ILOG JRules can also do this

by Gaurav Shrinivas /

• ##### Re: ILOG JRules can also do this

by wht wht /

• ##### Sounds interesting for a variety of uses cases - RSS, security detections?

Your message is awaiting moderation. Thank you for participating in the discussion.

This sounds pretty interesting and I can see an event processing query language amd framework like this being useful in a variety of use cases beyond financial or high volume. Imagine using it to consume thousands of RSS feeds and scan for patterns and do useful things with that info, or perhaps scanning incoming web requests in memory and detecting denial of service attacks or simply upsurges in requests/minute and being able to respond to that in smart ways.

Very cool!

• ##### Market Surveillance

Your message is awaiting moderation. Thank you for participating in the discussion.

Yummy!
And real-time risk management.

Regards,
Horia

• ##### ILOG JRules can also do this

Your message is awaiting moderation. Thank you for participating in the discussion.

I will have to download Esper and take a look, but ILOG Rules (C++) and JRules have been used for realtime event processing (both in telco and finance) for a number of years. For example you can write IRL code such as that included below (for event correlation and filtering). The RetePlus algorithm in JRules is an optimized pattern-matching mechanism that supports temporal reasoning. It is also possible to use the JRules tools to built domain-specific languages for business users that generate the code included below. JRules can reason on unmodified POJOs as well as XML data.

This rule detects that two alarms with the same Communications probable cause have occurred on two pieces of equipment which are connected together, and synthesizes a new alarm on the connecting link. In this rule the handling of partial order is illustrated. The id > alarm1.id test is here to break the symmetry between e1 and e2, and thus avoid therule being fired twice.when {	alarm1 : event Alarm(e1 : equipment;	   perceivedSeverity != Cleared;	   probableCause == Communications);	alarm2 : event Alarm(e2 : equipment;	   perceivedSeverity != Cleared;	   probableCause == Communications;	   e2 != e1;	   id > alarm1.id;	   this before[-10, 10] alarm1);	link : EquipmentLink(isConnectedTo(e1);	isConnectedTo(e2));}then {       insert event Alarm(link, link.newCorrelatedAlarmId(), Equipment, Major);}

Or this rule, which does event filtering:

This rule suppresses an alarm and its clearing alarm if the clearing alarm occurs within two seconds after the initial alarm.when {  alarm1 : event Alarm(perceivedSeverity != Alarm.Cleared);  alarm2 : event Alarm(equipment == alarm1.equipment;    id == alarm1.id;    perceivedSeverity == Alarm.Cleared;	this after[0, 2] alarm1);}then {	retract alarm1;	retract alarm2;}

Thanks for the link to Esper!

Sincerely,

Daniel Selman
(ILOG)

• ##### Re: ILOG JRules can also do this

by Alex Vasseur /

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Daniel
Thanks for your comment. I believe ILOG JRules and more generally Drools or Jess or any Rete based production rule engine can indeed be used to address part of what CEP coins. Especially triggering by correlating events, possibly including a temporal relationship between events as your sample shows.
I think ISphere folks have written about "delta dataflow" algorithms beeing a generalization of Rete ones.

That said I believe the ESP side - Event Stream Processing - is a different beast, where what matters is the "S" for Stream. In this side of the Esper engine we provide several language facilities to build expressions using time not for temporal relationship (happened before etc) but for sliding window.

As an example, this makes it very valuable for computing things like volume weigthed average (VWAP) of ticks - which would be (I believe, correct me if I am wrong) awfull using something Rete based like JRules. In Esper I'd wrote it as follow:

select * from StockTickEvent(symbol='BEA').win:time(3000).stat:weighted_avg('price', 'volume')

I believe beeing able to combine both CEP and ESP is a key value add of Esper. You can read more about how the two play together in our tutorials.

Alex

• ##### Re: ILOG JRules can also do this

by Mark Proctor /

Your message is awaiting moderation. Thank you for participating in the discussion.

I've been thinking on CEP and ESP, both are definitely possible in Rete - they will be a strong part of the work I'll do early next year. I have splashed some initial thoughts on ESP on the Drools wiki.

wiki.jboss.org/wiki/Wiki.jsp?page=EventStreamAn...

As Drools allows for querries we can easily provde ECA type syntax. Maybe we should be looking to join the projects together and build ESP/ECA on the drools platform?

• ##### Re: ILOG JRules can also do this

by Jin Chun /

Your message is awaiting moderation. Thank you for participating in the discussion.

Not to sound stupid, but I am literally drooling at the prospect of drools integration and Esper in general. This is the sort of stuff that is being pushed quite heavily right now, and having a first class OS alternative would be more than fantastic.

• ##### Re: ILOG JRules

Your message is awaiting moderation. Thank you for participating in the discussion.

Can any one help me with a sample "Suppress rule" in iLog JRules.

Regards
Gaurav

gauravshrinivas at yahoo dot com

• ##### Re: ILOG JRules can also do this

Your message is awaiting moderation. Thank you for participating in the discussion.

Daniel

my mail id is given.

• ##### Re: ILOG JRules can also do this

Your message is awaiting moderation. Thank you for participating in the discussion.

Friends,

Thanks as I was able to finish it well.

• ##### Re: ILOG JRules can also do this

by wht wht /

Your message is awaiting moderation. Thank you for participating in the discussion.

How many alarms could be processed in 1 second with ilog engine ?
100/s,or 1000/s?

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.