In 2002 David Luckham introduced the term Complex Event Processing (CEP). In papers several years later on the history of CEP, Luckham showed how although event processing has been a core part of distributed systems and simulations for decades, CEP is an evolution and not the same as, say, context switching. In fact CEP can best be summarised as:
Complex Event Processing is a technology for low-latency filtering, correlating, aggregating, and computing on real- world event data.
Over the intervening years, CEP was discussed within the context of SOA as a natural fit and at times called "a new physics of computing" or, as the "next big thing" in 2008 by WebSphere CTO Jerry Cuomo:
I've started to hear some people talk about this in the context of SOA, and I really believe it's the next big thing in SOA, and that's event processing. I don't know how much you're hearing about this, but certainly we're taking it very seriously. I see a spectrum of event processing. There are types of event processing that we are doing very well today on the simpler notion of event processing.
There have been several companies offering and developing CEP products or open source projects, such as Esper, Drools Fusion, Oracle and Sybase. At the same time we saw people talking about Event Driven Architectures (EDA) and Event Stream Processing (ESP), building on the concepts of events and CEP, with the likes of Gartner and Oracle even suggesting that the combination of CEP and SOA was the genesis of the next generation of SOA, i.e., SOA 2.0, though that didn't go down too well with the wider industry. However, as Eric Roch points out recently, often the term CEP can make it difficult to sell the underlying concepts:
The term CEP and the idea of its complexity is further exacerbated by the uses cases often associated with the technology. The most common use case stated is algorithmic trading. Can you imagine going up to your CEO and saying you want to try CEP, it’s the cool technology Wall Street is using for algorithmic trading? If you are lucky you might get, “Why in the heck do we need that in our business?” - Or maybe just an aggravated, dismissive stare.
Eric agrees with Luckham's original analysis that there's really not a lot new behind the concepts underlying CEP, such as Message Oriented Middleware and rules engines. But even here there can often be far too many complex terms to cloud the arguments:
But when you throw in all the buzz words– complex events, event channels, inference engine, Rete networks, forward chaining, state machines, tuples, algorithmic trading it does sound like you need a PhD to make it work.
Fortunately over the past decade implementations have improved and tooling is now often a critical component:
CEP tools, especially the most mature ones, are model driven and fairly easy to use. You really don’t need to know the internals of an inference engine to write business rules. They are in fact easier to write than a bunch of nested if statements.
However, over the past couple of years the amount of discussions and hype around CEP have reduced. This could be because of a similar reduction in that associated with SOA. It could be because CEP has not really delivered on the original promise. Or, more likely, it is simply because as Luckham pointed out, events have been a part of software systems for decades and now CEP is just a core or natural part of SOA implementations and people simply take it for granted.
Of course with the advent of Cloud and the push to take existing software investments there, we have seen discussions on moving SOA into the Cloud and of course that means all things associated with SOA, e.g., CEP. However, it seems that this may not be as simple as it may appear initially as Colin Clark reported last year:
Perhaps it’s because that the current incarnation of CEP products don’t lend themselves well to grid deployments. None of the current crop of products include a message bus or cache, something that’s necessary for grid deployment. And because this crucial component is missing, chances are that the products were never really architected for the “I don’t care where a message is coming from and I don’t care where it’s going” kind of world. The kind of world we are increasingly living in. The kind of world that readily lends itself to cloud computing.
As mentioned in a separate article, it is possible that some of the inherent assumptions behind CEP, such as low latency, or the "complexity" of current implementations is an impediment in using them in the Cloud. Or maybe CEP will be as critical to Cloud as it has been to distributed systems over the past 5 decades? So are you using CEP in your SOA applications? Is it a core part of your SOA infrastructure? Are you considering using it within the Cloud and if so, do existing implementations match your requirements or do we need something more Cloud specific?
Community comments
No internal takers for CEP yet
by Faisal Waris,
Re: No internal takers for CEP yet
by Mac Noodle,
Why talk about CEP in isolation?
by Jean-Jacques Dubray,
Accept events as they are...
by Dave Duggal,
Re: Accept events as they are...
by Dave Duggal,
Applicatio integration easier?
by Bill Burke,
No internal takers for CEP yet
by Faisal Waris,
Your message is awaiting moderation. Thank you for participating in the discussion.
As part of the extended SOA group, I have been looking for internal projects that have a needed for CEP but have not found any takers yet.
Maybe the issue is that project-level architects don't have CEP on their radar as a solution to some types of problems and so address similar problems with application development instead.
In our case, we want to tryout StreamInsight as it comes bundled with SQL Server R2 (i.e. is 'free').
I personally think that it is a very promising technology and maybe more useable as part of application (rather than middleware) as noted by another poster.
I think Microsoft should untether StreamInsight from SQL Server to make it more useful for general application use.
Why talk about CEP in isolation?
by Jean-Jacques Dubray,
Your message is awaiting moderation. Thank you for participating in the discussion.
"events have been a part of software systems for decades and now CEP is just a core or natural part of SOA implementations and people simply take it for granted."
Even though I agree with that statement, I am not sure it is useful to state it that way, not it is useful to carve concepts like CEP out of the Service Oriented Computing model.
I'd rather ask the question in an another way: Are Events part of a Service Oriented programming model? An event being the occurrence of a state, I hope by now everyone would say yes. The next question would then be do you need to detect and respond to combinations of events? again, I hope everyone will say yes.
The last question is do you need an engine that is configured to detect and invoke actions in response to complex event combinations? I would say, it depends.
Personally, I would rather focus on the question of what is a truly "Service Oriented" computing model. How do events relate to that model? Resources? Resource lifecycles? Processes? ... How can this model be deployed in the Cloud?
Answering that question could prove to be far more constructive than identifying specific concepts like CEP that apply to a specific (often very limited) class of problems. In essence such concepts are just a projection of a service oriented computing model.
Accept events as they are...
by Dave Duggal,
Your message is awaiting moderation. Thank you for participating in the discussion.
Thanks for bringing together these various threads on events. I believe the volume has been turned down on middleware in general, but events are an enduring concept that are perfectly suited for Service Orientation (and Resource Orientation) as JJ notes.
While I agree with Eric that "Algorithmic Trading" is a hard concept for a business-user to wrap their wrap their heads around, we use a related event/message methodology for situationally-aware enterprise apps - optimize business processes with context-enriched 'services' (right data/capabilities to right person at right time).
Our approach is aligned with William's and Faris's comments - events embedded in a unified authoring and run-time environment.
We treat every interaction as a discrete event, which we correlate to externalized rules to dynamically generate system payloads at run-time, sub-second. In this way we bridge the divide between OLAP and OLTP with analytics and execution in one.
It's low latency, includes an integrated ESB and provides a development environment that abstracts underlying complexity so developers don't have to understand or manage low-level implementation details.
We presented a paper at the Worldwide Web Conference / WS-REST workshop this spring (slidesha.re/g7Dprj).
Applicatio integration easier?
by Bill Burke,
Your message is awaiting moderation. Thank you for participating in the discussion.
When I think of middleware and cloud together I think of something like Amazon AWS. A generic CEP engine that is configured to process feeds and output feeds. This makes me think that it would be oodles easier to configure CEP as part of an application than as a generic cloud service. With an application your integration work can mostly be done more efficiently in code. With a generic engine, you'd have to define the incoming and outgoing feed and map the feed, via configuration data, to the CEP event processing mechanism. Sounds like a lot of work. Am I making any sense?
Re: Accept events as they are...
by Dave Duggal,
Your message is awaiting moderation. Thank you for participating in the discussion.
William - Thanks! Our website www.ideate.com goes into some detail, but here are a few other links -
“Putting Work in Context” (scr.bi/hcOyFh)presented at Coordination, Collaboration and Ad Hoc Process (CoCoA) workshop
Dr. Robin Bloor of the Bloor Group wrote a post “The New Kid on the Block” (bit.ly/gnZ8N2)
Forrester gave us a mention as an interesting new technology to watch in their 2011 Wave Report on “Dynamic Case Management” report (bit.ly/fWIjng)
I contributed a chapter, “Social Collaboration + Lean Integration = Enterprise Agility” to a new book on Dynamic Business Processes, “Taming the Unpredictable”, (futstrat.com/books/eip11.php) published by the Workflow Management Coalition (WfMC)
The SOA Advisory, ZapThink, wrote a whitepaper on our system, “Facilitating Emergent Processes” (bit.ly/qk2PWp)
Cheers,
Dave
Re: No internal takers for CEP yet
by Mac Noodle,
Your message is awaiting moderation. Thank you for participating in the discussion.
Sadly, that is the problem with most of Microsoft's server products. They try to be all things to all people. Why is SQL Server the ESB/ETL/Messaging/RDMBS/Haddoop Router/Document DB/App Server? Just let it be a good database. We should not need to install SQL Server just to use those other things. Plus, they cause DBAs to have to get involved. :(