InfoQ Homepage Presentations Managing Data in Microservices
Managing Data in Microservices
Summary
Randy Shoup shares proven patterns that have been successful at Google, eBay, and Stitch Fix. Shoup covers managing data, the need to isolate a microservice's data store behind the service interface, using events as a first-class tool in the architectural toolbox, techniques for service extraction from a monolithic database and much more.
Bio
Randy Shoup is a 25-years veteran of Silicon Valley, and has worked as a senior technology leader and executive at companies ranging from small startups, to mid-sized places, to eBay and Google. He is currently VP Engineering at Stitch Fix in San Francisco.
About the conference
Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.
Community comments
Data/persistence is always the hardest part
by Richard Richter,
Re: Data/persistence is always the hardest part
by Randy Shoup,
Event sourcing is the key part missing
by Dmitry Yegorov,
Re: Event sourcing is the key part missing
by Randy Shoup,
saga presentation
by mathys dikmans,
Re: saga presentation
by Randy Shoup,
Data/persistence is always the hardest part
by Richard Richter,
Your message is awaiting moderation. Thank you for participating in the discussion.
Very nice overview of the ways how to move from shared DB to true microservices. There are many talks about microservices, but too few tackle the data problem. (The same happens with CD and versining data schemas. It just shows that data evolution is perhaps the hardest part of it all.) I'm really glad to see approach 2 mentioned for join problem - although there are no details (understandably).
Couple of broader topics wrapping it all up don't hurt at all. I like that "build right" vs "build twice" (or often even multiple times) view.
Event sourcing is the key part missing
by Dmitry Yegorov,
Your message is awaiting moderation. Thank you for participating in the discussion.
The very thought-provoking presentation. Thank you so much!
Absolutely agree that event sourcing layer is something missing in the set of typical building blocks. Having it in place would allow unifying all communication between microservices. A sourcing service would expose its domain model by registering read-only projections (similar to a RDBMS view). Any service consuming those projections would automatically be routed/attached to proper event stream to keep internal state in sync (pretty much like a remote materialized view).
It is also possible to completely remove the need for RPC-style API replacing each call with a pair of Request/Response entities registered in the layer. Thus a service willing to initiate some external processing would create a new Request entity causing NewRequestCreated event. A service consuming that type of Request would grab the event, do the processing, and create a new Response entity. The requesting service would then be notified with NewReponseCreated event. This approach is specifically beneficial for high-performance applications as things occur in a completely asynchronous manner.
Re: Data/persistence is always the hardest part
by Randy Shoup,
Your message is awaiting moderation. Thank you for participating in the discussion.
Thanks, Richard! For "materializing the view", the pattern is more important than the details, because the details will vary. Conceptually, you are computing the join every time the underlying data changes instead of every time it is queried.
Re: Event sourcing is the key part missing
by Randy Shoup,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi, Dmitri. Glad you enjoyed the presentation.
One could certainly imagine a microservice ecosystem where all interactions are asynchronous, and that would have many nice properties. The Reactive Manifesto outlines in detail how this can work, and there are some domains where this is perfectly appropriate.
In practice, at the places I have worked, we have always had a combination of synchronous and asynchronous interactions. Some user interactions are just more naturally modeled by a synchronous call.
Take care,
-- Randy
saga presentation
by mathys dikmans,
Your message is awaiting moderation. Thank you for participating in the discussion.
The talk about saga's mentioned during the presentation can I view it somewhere online?
Thanks!
Re: saga presentation
by Randy Shoup,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi, Mathys --
Sure, the best talk on sagas is by Caitie McCaffrey (@caitie on Twitter):
Distributed Sagas: A Protocol for Coordinating Microservices (www.youtube.com/watch?v=0UTOLRTwOX0)
Take care,
-- Randy