Bindings, Platforms, and Innovation
This presentation focuses on the Internet and separating myth from fact, history from the future, and the mundane from the imaginative. Bob Frankston presents a vision of what could and should be.
Tracking change and innovation in the enterprise software development community
Posted by Floyd Marinescu,Thomas Bernhardt on Oct 12, 2007 08:40 AM
Esper (which version 1.0 was announced more than one year ago on InfoQ) is an event stream processing (ESP) and event correlation engine (CEP) that triggers actions when event conditions occurs among event streams - which can be thought of as a database turned upside down where statements are registered and data streams flow through. Event processing is a growing trend in the software industry, and several vendors have entered the market following a number of startups. Common use cases range from algorithmic trading, BAM, RFID, advanced monitoring systems and fraud detection up to a direct relationship with an SOA. InfoQ caught up with Thomas Bernhardt & Alexandre Vasseur on recent developments with the project.I think the fact that Esper plays a role in BEA's product helps the Esper project in a couple of ways. First, feedback gained is incorporated backWith the growth of this market space and the presence of multiple competing implementations, standardization could play some benefit. Thomas commented on the potential and background CEP language standardization:
into an improved Esper. Second, the BEA product raises overall awareness of CEP/ESP technology hugely and thus enlargens the mindshare and market.
Third, its a great testimony to how open, extensible, and enterprise grade ready Esper technology is. The Esper community and users base is really proud of that relationship.
The CEP community clearly sees CEP and ESP as complimentary, and recognizes that other approaches (i.e. baynesian or neural networks) also apply to CEP problems. In light of various approaches, and vendors not agreeing, the most relevant standard appears to emerge from work of the ANSI SQL standardization commitee extending SQL to provide "pattern matching in sequence of rows".Perhaps the most notable news related to Esper recently is the publishing of a performance benchmark kit and results, in mid-august:
There will for sure be further work on that early topic and standardization will likely go beyond ESP/CEP language standardization.
Esper exceeds over 500 000 event/s on a dual CPU 2GHz Intel based hardware, with engine latency below 3 microseconds average (below 10us with more than 99% predictability) on a VWAP benchmark with 1000 statements registered in the system - this tops at 70 Mbit/s at 85% CPU usage.Despite based on a rather simple use case, the publication of this benchmark work is aimed at shaking the industry, as it comes with a complete kit to replay the benchmark. An Esper event server is listening to remote clients sending market stock events over the network. The Esper engine is configured to compute volume weighted moving average of the feeds in real time over a sliding window of time or events.
The CEP market has been common place of vague information regarding performance and latency with every vendor throwing its figures in the press without any details at all. No comparative benchmarks exists in this area yet.The Esper team has also published the details of all its runs on its wiki and updated its product website with a performance section and performance best practices section. Another source of benchmarks may be coming from the newly formed STAC benchmark council, which aims to put out customer-driven benchmark standards for trading technology.
Vague performance information in this industry had already been criticized by Progress Apama and others . Here is a compilation from the Apama blog:* Skyler manages rates as high as 200,000 messages/second * Key feature: Coral8 handles thousands to millions of events per second * StreamBase extends performance leadership by processing over one million events per second with near zero latency * Aleri Labs breaks sub-millisecond latency barrierApama itself claims to be "a high-performance, scalable processing engine that can process thousands of events per second". Such claims could also be found in the BEA wording regarding their WebLogic Event Server announcement with inferior yet more precise figures: "As we come out of the gate, we're going to provide 50,000 complex events per second".
Those results seems to confirm that "hundreds of thousands" events per second is common and no exceptional in this area, and also show exactly how Esper performs on the given scenarios. It also gives valuable material to the user community to better assess performance instead of listening to random vendor FUD commonly throwned at disruptive yet affordable open source software.
Download the Free Adobe® Flex® Builder 3 Trial
Adobe® Rich Internet Application Project Portal
How Java Developers Can Write Great SQL
No comparison to kdb+ here, which is pretty quick - and has been applying vector processing to stock prices (among other things) for ages. e.g. select amount wavg price by date from trade where stock=`BT.L check out www.kdb.com/q for more info. i have no connection with kdb or 1st derivatives.
I believe Kdb+ (at kx.com by the way) is primarily a time series database. As such it does not have features for causality ("happened before" relation between events), and I expect it to be pretty hard to deal with "absence of events". By contrast Esper features both event stream (dealing with sliding windows of real time data stream) and complex events (happened before and other time guard constructs). The vwap benchmark that illustrates Esper performance is certainly a simple enough use case to compare Esper with a time series database, but it is far from giving a complete overview of what can be achieved with Esper event processing and continuous queries capabilities. This said kdb+ is an interesting piece of software and I believe this should be fairly possible to integrate Esper with KX kdb+ for f.e. event replay capabilities. Alex
unrelated to the kdb+ question above - i had a cause to use esper recently to measure application performance. On a current SOA project i was able to determine application latency and throughput by matching the correlation-ids being output by the various systems. adding a performance monitor with esper took about three hours from concept to finished implementation. good job guys.
sorry i forgot to mention - it required not a single application change.
Just got back from the Gartner CEP event and did not hear about Esper there (not a big surprise, not much talk about open source in general there). This article is dated, so I'm wondering what is going on with Esper? Has it gained any momentum? As far as benchmarks, it seems that with event processing, low latency and messages per second are stressed to the extreme. Performance in the micro-second range may be the cost of entry to play in the algorithmic trading space, but there are many, many more use cases for event processing where the events are measured in hundreds or thousands, not millions per second. At the show, even many trading companies mentioned that performance was not their main criteria for picking a vendor ...
This presentation focuses on the Internet and separating myth from fact, history from the future, and the mundane from the imaginative. Bob Frankston presents a vision of what could and should be.
This article explores the use of JBoss and jBPM to implement design solutions that effectively address the issue of orchestrating long running activities.
This presentation covers the use of graph databases as an optimal solution for data that is difficult to fit in static tables, rapidly evolving data or data that has a lot of optional attributes.
This session introduces Real Options and shows how it can help in running your project. Real Options is a decision-making process that can be used to manage risk.
This article discusses the use of bindings on services and references (including the instance of non-configured bindings) as the means to implement SCA communications in a Web and SOA environment.
After a short introduction to DSLs, Scott Davis plays with the keyboard showing how to approach the creation of a DSL by typing working snippets of Groovy code that get executed.
IBM Rational and InfoQ present, Scaling Agile with C/ALM, an eBook showing organizations how to become “finely tuned software delivery machines” by enabling team integration and scaling.
Amanda Laucher presents a real life enterprise application written in F#. She shows actual code snippets, explaining design decisions and suggesting how to use some of the F# constructs.
5 comments
Watch Thread Reply