BT

InfoQ Homepage News Mistakes and Recoveries When Building an Event Sourcing System

Mistakes and Recoveries When Building an Event Sourcing System

Bookmarks

When Nat Pryce and his team started building a new system based on an event sourced architecture, they made a couple of significant mistakes in the design, but managed to recover from these mistakes with an ease that surprised them. In a blog post, Pryce describes the mistakes they made and the factors that made it possible for them to refactor the architecture and recover from their mistakes.

Their first mistake was persisting the event history together with persisting a view of the current state of the corresponding entity. The current state was not a projection and updated from the events, but rather was updated by the command handler that recorded the events. This introduced two problems: the state of an entity could not be rebuilt from the recorded events, and managing migrations in the relational model they used for the current state proved to be a significant overhead.

Pryce, co-author of the book Growing Object­-Oriented Software Guided by Tests, admits that keeping the two persistence mechanisms together was in a way missing the whole point of event sourcing. The reason for the mistake was that they came up with a design that the team was comfortable with, without reflecting over the mismatch when comparing with recommendations in the event sourcing literature. They continued with this design until the difficulties clearly outweighed the benefits. They then had a technical retrospective and agreed on moving to a canonical event sourcing design.

Their next mistake was a confusion between event-driven and event-sourced architecture. In an event-driven architecture, components perform tasks in response to received events, and emit events to notify about changes in state. In event sourcing, state changes are recorded as events and the current state of an entity is calculated from all events related to the entity. This confusion led to a design where a component both recorded events in the history, and triggered activities in other components. They realized their mistake when they had to implement logic in events to distinguish between a) reading an event and reacting on it, and b) reading an event to know what happened in the past.

This confusion also led to a design where they used the event store as a message bus. They started to emit notifications to enable for components to keep their projections up-to-date, which meant they used the event store both for the event history and for transient communication between components. This gave them an event store that included technical events that had to be filtered out of the history before being displayed to consumers.

The last mistake Pryce describes was the usage of an HTTP interface for reading and storing events in the event store. This prevented the team from processing events in ACID transactions, instead forcing them to build other mechanisms in an attempt to mitigate this.

Fortunately, they discovered all their mistakes early on, before the event history was affected in their live system. The HTTP interface was replaced with direct database connections for their command processors. They stopped using notifications and went back to using REST for passing data between components. Finally, they moved away from updating entities’ current state in command handlers. Instead, the state is computed from the event history when an entity is loaded. They still use a projection of the current state of events, but this is seen as a read-through cache, purely for optimization.

Pryce concludes by noting that although they made significant changes to the architecture, the changes were straightforward, and he points out that the reasons for this are orthogonal to event sourcing.

The application has a ports and adapter architecture which makes it easy to change an implementation when it’s hidden behind a port or an adapter interface. They have extensive functional tests, written to take advantage of the ports and adapters architecture. This way, the technical architecture is segregated from the implementation of the functional behaviour, and this simplified when the changes of the architecture were made.

For Pryce, it’s inevitable that mistakes will be made when you adopt an unfamiliar architecture style in a system. He believes though that the ports and adapter style allowed them to adopt event sourcing despite their lack of experience, but also to recover from their misunderstandings when building the system.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Transitioning to an event store while feeling comfortable

    by Guy Pardon /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hi,

    Nice read!

    This article outlines an alternative transition path towards event storage - what do you think?

    Guy

  • Re: Transitioning to an event store while feeling comfortable

    by Jan Stenberg /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hi Guy,
    I've read the article and I have some objections or comments.

    For me, DDD is mature and well understood. I don't agree with the high risks mentioned. Instead, I think that in complex domains (where DDD should be applied) the risk is in not using DDD.

    Event sourcing is not mandatory when using DDD, you may very well use DDD and a relational DB.

    The events stored when using event sourcing are for me internal events, they should not be publicly available. To inform other services about what has happened I will use Domain Events.

    The solution mentioned, updating an entity in a DB and storing an event in an event table within the same transaction, is the Outbox pattern. You can then publish the stored events using messaging, Atom feeds or some other technology.

    If you want high throughput when using event sourcing, you don't do inserts into a database, you should use an append-only event log, which is much much faster than a relation DB.

    You don't query an event store; you use projections to create read models you run your queries against (CQRS).

    Thanks,
    Jan

  • Re: Transitioning to an event store while feeling comfortable

    by Guy Pardon /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hi Jan,

    Thanks for the feedback!

    DDD is well-understood by experts like you, but not (yet) by everybody :-)

    I agree that event sourcing is not mandatory. Whether or not domain events are stored versus internal events: that depends on the point of view it seems, I think Greg Young stores (stored?) domain events (but I could be wrong).

    With" append-only log" you mean a file-based solution? We went through a great deal of effort to replace files with DBMS tables for cloud-readiness.

    Thanks
    Guy

  • Re: Transitioning to an event store while feeling comfortable

    by Jan Stenberg /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I think it's quite easy to get a basic understanding of DDD, there are introductory books as well as practical books. There are also a lot of training available.

    I've seen some confusion about different types of events since event sourcing became more discussed. For me Domain Events are used for communication between Bounded Contexts. Internal Events in event sourcing are used within a bounded context.

    You don’t have to implement a file-based solution yourself. There are products available that implement append-only logs, like EventStore, Axon, Kafka, etc.

    Thanks,
    Jan

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.