For problems that have to do with time like querying over historical data Complex Event Processing, CEP, can be very useful, Greg Young suggests in a recent presentation.
Greg describes event processing as a method of analysing streams of data about things that has happened and deriving a conclusion from them. One example is temporal correlation queries used when you want to correlate things that have happened at different times, e.g. finding users that said the word “presents” within one minute of saying the word “birthday” in a twitter stream or finding patients in a clinical trial that has reacted in certain ways at different times during the trial period.
For creating queries Greg uses a query language implemented in JavaScript that works on top of event streams. Writing queries in JavaScript means a web browser can be used for writing, and also for debugging.
This query language is included in Event Store, a type of NoSQL database built according to the concepts of an Event Source. In event sourcing, instead of representing state directly, state is represented as a series of facts, something that has happened at a specific point in time. Storing data this way means that no information is lost since every change of state is stored, not just the current state.
Storing facts or events creates a kind of time machine making it possible to go back in time and look at how things were historically, e.g. creating a new report today and run it as if time is one month back.
Another use case is looking at predictions and estimating the predictivity capability; running a predictive model as if it is one month earlier, predicting outcome for today and comparing the result with the real outcome.
Event Store is an open source product, licensed under a 3-clause BSD license, first released in September 2012.
Greg Young, lead architect for the Event Store, is an independent consultant famous for having coined the term CQRS.
/filters:no_upscale()/sponsorship/topic/995382dc-b781-4f0b-b74a-413e61a30927/CurityLogoRSB2023-1675420245128.png)
/articles/billions-messages-minute/en/smallimage/LOGO-1679471830613.jpg){kind=logo}
/presentations/event-tracing-monitoring/en/smallimage/glen-meiler-s-1671178393152.jpg)
/presentations/refactoring-rust/en/smallimage/qcon-plus-s-1679694299933.jpeg)
/presentations/aws-apache-airflow/en/smallimage/qcon-plus-s-1679694143127.jpeg)
/articles/ci-cd-observability/en/smallimage/logo-1679470476672.jpg){kind=logo}
/articles/visual-thinking/en/smallimage/LOGO-1679339789328.jpg){kind=logo}
/articles/framework-architectural-decisions/en/smallimage/logo-1679060242842.jpg){kind=logo}
/articles/asp-dotnet-minimal-apis/en/smallimage/logo-1678972581766.jpg){kind=logo}
/presentations/staff-plus-open-source/en/smallimage/alex-porcelli-s-1679038616213.jpg)
/presentations/java-compiler-intermediate-representation/en/smallimage/chris_seaton-s-1679038354209.jpg)
Community comments
Still too much mis-understanding of event processing
by peter lin,
Re: Still too much mis-understanding of event processing
by peter lin,
Re: Still too much mis-understanding of event processing
by Greg Young,
Still too much mis-understanding of event processing
by peter lin,
Your message is awaiting moderation. Thank you for participating in the discussion.
The talk is interesting, but it's rehashing a lot of common mis-understanding of event processing. Event processing was never about "now". Just because some CEP products sold that as "the story", the history of event processing goes back at least 4 decades. The way Rich Hickey explains Datomic is a better explanation of this "class of problems" with grounding in formal logic.
Re: Still too much mis-understanding of event processing
by peter lin,
Your message is awaiting moderation. Thank you for participating in the discussion.
Agree 100% CEP is past "sell by" date. It wasn't "new" when it came out, just the sales hype tried to paint it as new. Most of the first generation CEP couldn't really be used for what you call "simulated playback" where different scenarios are used to try out different "what if" scenarios. They were only really good for literal playback of a sequence of events, though even then the issue of partially-ordered versus totally-ordered sets reared it's ugly head.
The biggest limitation is that most of the CEP products early on only covered sliding windows. They ignored 95% of the challenges related to temporal logic and temporal database. At some point, people want to playback the events with different models and scenarios, but most of the products didn't care about saving those events to a temporal database. It was up to the developer to figure out how to efficiently store temporal data. It's understandable most of the products didn't tackle temporal database problem, since "the field" made very little progress after tsql2 was rejected.
Now that the hype around CEP has died down, hopefully more fruitful progress can be made towards solving the broader problem of how to reason over temporal data and how to efficiently store/query it.
Re: Still too much mis-understanding of event processing
by Greg Young,
Your message is awaiting moderation. Thank you for participating in the discussion.
<< Greg Young here
I think you might have misunderstood the talk or maybe in time constraints it was not explained well enough the query language is over HISTORY that transparently moves into now.
Greg