Tracking change and innovation in the enterprise software development community
Create REST
-enabled Web Apps
with Project Zero
from IBM.
QCon San Francisco
Nov 19-21:
+ 80 sessions & tracks
Register now
and Save $400!
IBM WebSphere
Virtualize apps
to lower costs,
Increase agility
Learn more!
Terracotta:
Scale your
Apps w/ease
Offload the DB:
Learn More!
Posted by Ian Roughley on May 21, 2007 10:06 AM
At Java One Thomas Bernhardt and Alexandre Vasseur explained the concepts of event driven application servers and the Esper project.Event driven application servers are a new category of servers, proving a runtime and supporting infrastructure services (transport, security, event journaling, high availability, connectors, etc.) to servers designed to be able to process over 100,000 events/sec. As well as event processing, event driven application servers are able to combine event information with long lived historical data (usually obtained via relational database queries) and performing temporal correlation and matching on the event streams.
There are two concepts that make event systems different from messaging system:
Fully featured application server are still a few years away, but developers can implement event driven architectures in stand alone applications, Java EE applications and Spring applications today using Esper from Codehaus. Esper version 1.0 (as reported by InfoQ) was released in June 2006, and is a lightweight, embeddable open source implementation of both ESP and CEP.
Integrating Esper into stand alone applications is easy. The steps are:
Events can be represented as java objects, XML or maps, and as they flow through the system, statements will be evaluated and listener logic executed.
The Esper query language provides a rich syntax allowing complex temporal logic to be expressed, and includes features such as:
Even though event driven application servers are a few years away, Esper is ready for production use today. Integrating Esper into your applications is easy, and will allow you to provide features that will anticipate business and customers needs in real-time.
Create a photo album application with Project Zero and REST design principles
The Agile Business Analyst: Skills and Techniques needed for Agile
Rainmaking - IBM's software virtualization strategy (Jerry Cuomo CTO blog)
I took a look at esper's pattern matching implementation last week and was wondering if anyone test espers with a dataset of 500K or 1million objects? I should state that I work on rule engines and have been studying RETE for the last six years. Looking at esper's node classes and design, my guess is espers will likely run into scalability issues with 250K objects or more. The way joins are performed is going to lead to inefficient pattern matching, which will lead to huge CPU usage. A mature RETE engine can handle 1million facts without any problems. This is from first hand experience. peter lin
Hi Peter, yep we have a benchmark available out of the RFID domain that features tracking 3000 assets as they move from zone to zone detecting when assets in a group split between zones. We have been able to get 110,000 events per second, sustained, on a laptop with DualCore 2.4 GHz. Details are on the JavaOne slides downloadable from our site. We also have various microbenchmarks in different examples and regression tests that we run as part of our regular build that process large data sets. Note there is no industry standard benchmark at this time. We are planning to have a performance test suite available that can be used for capacity planning, hopefully soon. Esper is designed to process large volumes of streaming data on a continuous basis. It indexes join fields and is therefore able to handle patterns involving joins rather well. Cheers Tom, of the Esper team
Even if the join nodes index the data, the order of the joins (aka query plan) can make a huge difference. Depending on the query and execution plan, the delta between an optimized query plan vs non-optimized can range from 3-4 orders of magnitude. this is from first hand experience comparing non-RETE rule engine vs RETE rule engine on a dataset over 500K objects. by rule engine, I mean an engine that performs pattern matching regardless of the algorithm used to compile the query statements. peter
Thanks to Ian and InfoQ for our JavaOne session coverage! If any JavaOne attendee want to send us feedback on our session please email us (avasseur on codehaus).
I just want to stress out that Esper and ESP/CEP is about continuous streams and real time filtering, aggregation and pattern detection among event streams. It is not a rule engine and I don't think we can really draw a comparison there (and this is not what the industry seems to be at neither).
At that stage I believe most of the users are more interested by the kind of problem EDA and ESP/CEP enables them to solve rather than raw performance.
Peter, I thus assume all systems that have to deal with 500K "object dataset" (unclear what we are talking about there) require some optimization. That said when it comes to ESP/CEP like Esper there are 4 things you need to consider - and I 'd be please to hear your comments if the same are key performance evaluation criteria in a rule engine (which I again argue is a different beast for a different purpose):
I believe our RFID sample that Tom commented illustrates some of this and gave our audience figures to remember at the end of a 1 hour session (2000 statements, 100K event/s on utility laptop, less than 2% matching ratio). I'd be happy to report some more on performance if you want to submit a use case that you would like us to consider for a benchmark (we are also looking at growing the commiter base if you want to bring some of your knowledge on that!).
Did you wonder how (if at all) you could implement the RFID asset tracking example we have using something like Drools (RETE based rule engine, I don't know about its optimization)?
It might be you'll come up with another use case that can actually be solved by both a rule engine and a ESP/CEP engine, this is likely an overlap area (yes there are some) and there at the border line, you may have deuce or one solution may clearly outperform the other one as it will be just the right tool for the task. It 'd be nice to start such an exercice.
Alex (Esper team)
you are right the query plans are important for the regular inner joins as well as outer joins, and they each require different execution plans. As our software is open-source, you can look at the code implementing the query analyze down to the execution, in packages net.esper.eql.join.plan. -Tom (N)Esper rocks
I was thinking of real-time trading systems, which handle a constant stream of transactions. The input varies between trading firms. Some firms handle over 10K transaction/second, other handle less than 1K/second. To me, CEP and ESP is just a new term, but it's the same old stuff that Event driven architecture (EDA) have been doing for decades. Say we have a system that handle 2K transactions/second.
2000 X 60 seconds = 120,000 / minute
120,000 x 60 minutes = 7,200,000 / hour
Orders generaly will stay in a OMS system for several hours from the time an order is sent to the time the order is closed. The RFID example isn't all the interesting to me. I've worked on real-time pre-trade compliance systems have to have handle thousands of diversification rules. They include government regulations, aggregations, calculating risk (exposure) and rating. These are real-time with transactions which are about 3.5K messages. Given the constantly shifting data, calculating the risk of a mutual fund using price aggregates, ratings, and aggregate ratings is rather challenging. Each calculation is rather simple by itself. It's keeping up with the constant stream. an example rule in CLIPS format might look like this.
(defrule sector_wieght
(transaction
(ticker ?ticker)
(customer ?customerid)
(quantity ?quan)
(price ?price)
)
(security
(ticker ?ticker) ;; this joins on transaction.ticker
(sector ?sector)
(country ?co)
)
(customer
(id ?customerid) ;; this joins on transaction.customer
(portfolio ?pid)
)
?aggr <- (aggregate
(portfolio ?pid) ;; this joins on customer.portfolio
(sector ?sector) ;; this joins on security.sector
(country ?co) ;; this joins on security.country
)
(bind ?newWeight (calculate-weight ?aggr ?quan ?price) ) ;; this calculates the new weight and binds it to a variable named "newWeight"
(test (> ?newWeight 15%) )
=>
;; send out a compliance violation message to the OMS, which stops the transaction
)
If I wasn't so lazy, I'd translate it to SQL syntax, but hopefully the example provides some context.
I see quite a few CEP and ESP vendors tell people to use their system for algorithmic trading. Some try to make sound like it's new, but it's basically what OMS system have been doing for over a decade. Many of the existing system have been using RETE to do real-time event processing. It's just most people outside don't know about it. Then again, most of the firms using RETE to do EDA keep quiet, since they consider it a strategic advantage. The military has been using RETE to do event filtering for command control systems for over a decade. You can imagine how much data a military command control system handles per second with radar and satellite data streaming in 24/7.
peter
Looking at the QueryGraph class, I get the impression, the query plan processes the sql where clause in the order it was written. It also appears a join node in the query graph can take a list of objects on either side. Is that an accurate interpretation? peter
No the order of the expressions in the where clause does not really impact query plans, you got that wrong. Esper builds an internal model based on the where and outer join criteria and works from there. -Tom
The example CLIPS seems really hard to understand though. A nice SQL-like syntax comes in handy. -Tom
thanks for clarifying that. So when the "where clause" is executed, where does it begin? Does begin with the object with more conditions or one with less? if I have time later tonight I'll try to translate that rule to Sql. one could break that up into multiple sql queries or one with several subqueries. peter
Here is a rough translation into Sql. Obviously, this isn't functional code, but it should give you an idea. You could combine the 2 queries in 1 query with a subquery.
select
cus.portfolio, sec.sector, sec.country into, trans.quantity, trans.price aggregate
from
transaction trans, security sec, customer cus, aggregate aggr
where
trans.ticker == sec.ticker and cus.id == trans.customer
select
trans.*
from
aggregate aggr, transaction trans, customer cus
where
trans.customer == cus.id and trans.ticker == sec.ticker and
sec.sector == aggr.sector and cus.portfolio == aggr.portfolio and
sec.country == aggr.country and aggr.newWeight > 15%
This kind of rule is actually fairly simple in the pre-trade compliance world. Other rules are more complex and calculate risk across the firm or a large mutual fund.
peter
I was just reading over the docs on esper website and had a question. In the "select avg(price) from StockTick.win:time(30 sec) where symbol='BEA'" example, does Esper run the query every 30 seconds? If so, does that create a thread to poll? The kinds of event processing I've dealt with didn't have a sliding window of x seconds. Instead, it's more like "when an optimal for the transaction is found, alert the trader". The other kinds of event processing I'm familiar with also don't use a sliding window. Should an event satisfy the conditions, the trigger is executed. If esper is using a polling mechanism, I question is "event processing" accurate? A polling mechanism is more like a batch process with a short latency between executions. peter
Peter, you raise a good question and again this is one the key difference I am urging you to make when putting RETE rules engine and ESP/CEP engine like Esper in the same area while they are not. Time is a first class citizen in Esper, as well as sliding window (governed by time or number of events etc). On the statement you take from our doc, there is no polling at all - it is a continuous query over a sliding window and you literaly get something happening in the engine (that filters out or do the aggregation computation and/or produces an output event) everytime an event flows in. Alex
Thanks for clarifying that. I'm not sure I get it, but I'll attempt to summarize what I "think" I understand. Esper doesn't use a thread to execute queries periodically. Instead, when an event enters the system, it goes through a set of filters, which are the compiled sql statements. The sliding window then defines a condition which says "if x condition happens at a max/min of x time, then do something". The kind of real-time processes I've worked with are OMS related. This means there is no max/min sliding window. There are thousands of transactions in the system and all of them have a different expiration time. In a system like an OMS, it can't do win:time(30 sec) because that doesn't make any sense. A sell order might say mininum price of xx.xx dollars and x shares. If any buy matchs that price and shares, it should go through immediately. Waiting for 30 seconds could mean someone else fill that order. The more I look at CEP/ESP, the less useful it becomes. When i compare RETE to Esper, I'm only looking at the compilation of the query. RETE provides one of the most efficient ways of compiling a query into an optimized query plan. Those who say RETE is not a good fit for EDA either A) have never bothered to study RETE or B) have misconceptions about what RETE is. I see alot of people saying "RETE is wrong for EDA, CEP, ESP" and go on to show a sql like query to prove their point. The first part of RETE is compiling a statement into an optimal query plan. The second part is the runtime indexing. One can apply RETE compilation and forget the runtime, or adapt the algorithm for temporal execution. RETE compilation has nothing to do with whether "time is a first class citizen". It is just an efficient method for compiling statements into optimal relational queries. Just because some rule engines don't support temporal logic, does not mean RETE is wrong or inappropriate as some commercial vendors are saying. To my knowledge, Esper has never made such a claim, but I have seen some commercial CEP vendors make those kinds of statements. For those who forget, temporal logic is one of the areas that AI and expert systems have pushed. Many of the advances in temporal logic came from AI research. i'm curious, how does a CEP system handle removal of events. in rule engine terms, when an event is retracted. I've tried to read up on CEP/ESP the last few days and the literal seems rather thin compared to the mountain of literal on pattern matching and rule engines. my bias 2 cents peter
There is some ongoing EDA reference architecture work going on at OMG so you might be interested in adding your bits there. As I said I think CEP and rule overlap some, ESP far less, and all are part of EDA - and I am bias as well. I'd be again happy to consider a use case with real working code. If I ask an ESP/CEP engine to handle the OMS side of thing, might be a rule engine will do a better job. If I ask a rule engine to detect a tripple bottom pattern on a stock tick, might be an ESP/CEP engine will do a better job. It could be the RFID sample we have in our Esper JavaOne slides can be solved both ways. If one want to give it a try with his favorite rule engine please do. The statements are in the slides and all the running code + demo GUI is in our SVN. By the way, it happens a very similar thread on RETE 'vs/for' CEP/ESP was started here. Could you confirm you posted there as well on May 7 - ie Peter == woolfel? The posts from this pseudo look very similar if not the same... Alex
Yup, I posted several comments on jboss blog. In terms of OMS and real-time trading systems, I know of several that use RETE engines for that. One of my long term goals is to create a set of "realistic" standard benchmarks for rule engines. One of them is a pre-trade compliance scenario for processing transactionsets. I have some use cases in clips format, but it's not something the average developer is going to understand. If you want a realistic use case, i would suggest looking government regulations like 1940, 2A7 http://www.sec.gov/rules/final/33-7479.txt I've built pre-trade compliance systems using JESS, so I have a little bit of experience building real-time compliance for OMS. Most diversification rules require the system incrementally recalculate the aggregate as transactions come ine. The aggregates are basically multi-dimensional aggregates and vary between 12-20 dimensions. peter
Are you the same Alex that made this statement on jbossrules blog? "On the if Rete fits CEP/ESP I think making time and causality a first class citizen is likely to deeply impact any Rete algorithm implementation. Let's left aside clustering and near real time performance requirements, or joins to relational database and continous joins. There has been extensive research in the CEP/ESP field and also around Rete and I'd tend to argue if researchers haven't come up to a common implementation so far, this is likely because some walls were hit. Both play a key role in an EDA but I don't believe a one size fits all there." I'll be blunt here. Very few people understand RETE well enough to implement a high performance rule engine. There's maybe 2 dozen people who know RETE well enough to implement a high performance engine. Dr. Forgy, Gary Riley, Ernest Friedman Hill, a few researchers at iLog, paul haley and a few guys who worked at ART. the statement about enhancing RETE so that "time is a first class citizen" is untrue. In my blogged, I provide several detailed description of how one can enhance RETE to support temporal logic. The type of temporal logic used in business system is only a tiny subset of temporal logic used in AI. There isn't a consensus from AI researchers about the best way to handle temporal logic because the AI case is 100x harder to handle than the simple business cases. I have blogs that attempt to describe temporal logic and how it differs between AI and business rules. If you want an invite to my blog, email me at Woolfel AT gamil DOT com. peter
Alex How does esper stack up against what's outlined in "Coral8 Guide To Evaluating ESP Engines"? ? thanks
Yes - same Alex and no doubt at all on that for anyone as my first name, last name, pseudo, and affiliation to Esper are explicit in both. The fact that you don't point me to a real implementation or research papers but to some posts in a blog just makes my point: I would tend to argue convergence as not happened yet. In all case we are pretty happy with Esper performance thus far and so are our users, and if at some time we get smart enough to understand RETE (i.e find time to study it properly - which I haven't so far) and feel it 'd bring something to our users we'll certainly work on changing our underlying implementation. We'd welcome contribution on that if you are interested. Alex
Great link. Thanks for pointing to it. I think Coral8 did a great job at helping possible users evaluating ESP/CEP technology. That said in this paper you'll see a few thing:
Which makes me suggest that if you are conducting an RFI on ESP/CEP engines I'd be please to help to evaluate Esper in your own context.
A fundamental difference though is that Esper is open source.
Alex
By real implementation, what do you mean? JRules supports stream processing and so does JESS. If you want proof that RETE can handle stream processing in an application that uses Event Driven architecture, all you need to do is look at JRules and JESS. If you want a real world application, sadly I can't provide any. I do know of systems using RETE, but that code belongs those companies. Like I said in a previous comment, I do plan to build a "real world" compliance scenario for real-time trading systems, but I haven't finished it. More accurately, I don't have enough free time to implement a full application. I do know some firms are experimenting with ESP/CEP products to do hedge fund stuff. Actually, many firms have been building these types of systems since mid 90's. They aren't general purpose solutions. There's no point in re-implementing RETE, when you can use JBossRules for the pattern matching. Not only do you get an efficient RETE implementation, you get support for many first order logic concepts like existential, negation, forall and collect. I haven't read the spec for StreamSql, but from the examples I've seen so far, i don't think it is expressive enough to support FOL. Do you know if StreamSql supports existential, negation, forall and collection? peter
Just to put this in context, lots of us are really excited about Esper, for all the obvious reasons, but we need your help, help us convince management that this is viable contender to the entrenched vendors... Somewhat like what JBoss did to penetrate the enterprise. I'm not talking about benchmarks, that's for marketing. My app is my benchmark. Also, updating the license to something more friendly would be helpful. So here's the question again, how does it stack up against the coral8 guide and Stonebraker's rules ?
sorry for the late reply. In terms of the Stonebraker rules, here is some short answers (a) Keep the Data Moving Esper has no costly storage operation in the critical path, our tests results show extremely low latency (b) Query using SQL on Streams The query language implements SQL and providing extensions for event stream processing and pattern matching/CEP (c) Handle Stream Imperfections (Delayed, Missing and Out-of-Order Data) Some of the features that Esper provides particularly to deal with these issues are joins, outer joins, patterns, subqueries and data windows. (d) Generate Predictable Outcomes Yep we have worked hard to get deterministic and predictable processing under multi-threaded conditions (e) Integrate Stored and Streaming Data Esper allow SQL queries to be placed right within the query language and provides expiry-time or LRU caches. (f) Guarantee Data Safety and Availability Esper does not currently offer a persistance mechanism for events. We are working on a HA feature set. (g) Partition and Scale Applications Automatically, and We are working towards these goals in the HA feature set (h) Process and Respond Instantaneously. We achieve that through highly optimized filtering, query planning, indexing and execution and other optimizations.
Thin client paradigm characterized by web applications is a kludge that needs to be repudiated. Old compromises are no longer needed and it's time to move the presentation tier to where it belongs.
In this presentation filmed during QCon 2007, Jeff Sutherland, the creator of Scrum, talks about his visit at Google to do an analysis of Google's first implementation of Scrum.
In this article, Bryon Jacob and Chris Berry introduce AtomServer, their implementation of a full-fledged Atom Store based on Apache Abdera, which is now available as open source.
It is easy to think that virtualization applies only to servers. In reality the recent resurgence of the concept is also being applied to networking, storage, and application infrastructure.
In this article, Stefan Tilkov explains some of the most common anti-patterns found in applications that claim to follow a "RESTful" design and suggests ways to avoid them.
In this article, Adrien Louis and Marc Dutoo discuss the differences and relative merits of using orchestration vs. routing in a typical ESB setup, and discuss various implementation options.
Wayne Lund discusses batch processing, Spring Batch objectives and features, scenarios for usage, Spring Batch architecture, scaling, example code, failures and retrying, and the future roadmap.
Developer Jay Fields draws on his experiences as a ThoughtWorks consultant to describe effective user story estimation techniques.
23 comments
Reply