Microservices for a Streaming World

| by Jan Stenberg Follow 37 Followers on Mar 14, 2016. Estimated reading time: 2 minutes | NOTICE: The next QCon is in San Francisco Nov 5 - 9, 2018. Save an extra $100 with INFOQSF18!

Embrace decentralization, build service-based systems and attack the problems that come with distributed state using stream processing tools, Ben Stopford urged in his presentation at the recent QCon London conference

For Stopford, working with Kafka at Confluent, there are many good reasons for building service-based systems.  These include loose coupling, bounded contexts, ease of scaling etc., all of which allow us to build systems that can evolve over time. But by taking this approach we are inherently also building distributed systems, which brings its own complexity with issues around latency, failures and so on.

Two fundamental patterns of distributed systems Stopford describes are:

  • Request–Response as a way to decouple services, typically using REST, which works well for UIs and when asking questions.
  • Event-driven, characterized by asynchronous or “fire and forget” messaging, great for composing complex dependencies across services.

These can also be combined, using request-response for a REST interface and events for background processing.

Looking into asynchronous and event-based communication, e.g. using queues, Stopford sees this as a very simple model, and as long as only one message is pulled at a time the ordering of messages can also be guaranteed. This can scale to a certain degree while still retaining the ordering guarantee, but Stopford notes that at some point we will lose either availability or the ordering guarantee. Another disadvantage he notes is that messages are transient, thus lacking the possibility to go back in time and read old messages after failures.

Stopford believes that a better approach is using a distributed log as a service backbone, with Kafka being one example. Kafka is based on the concept of a Log, which is an append only data structure. This makes both reads and writes efficient, for reads it’s a matter of a single seek to a position followed by sequential reads and for writes it’s just an append.

Some advantages for microservices that a distributed log enable includes:

  • Always on, relying on a fault-tolerant broker, like Kafka.
  • Load balancing, with service instances each reading data from a broker.
  • Fault-tolerant, since services can fail over but still retain ordering of messages.
  • Rewind and replay, allowing for a service to return to old messages and replay them, e.g. after an error is discovered and fixed.

One problem not solved is keeping services consistent. After e.g. failures it’s hard to avoid duplicate messages (“at least once” delivery) making it necessary for services to be idempotent regarding messages they receive, logically creating an “exactly once” delivery mechanism. Stopford notes that this is not yet available in Kafka (but work is ongoing).

Martin Kleppmann addressed the service consistency problem in his presentation at the same conference.

Stopford’s presentation is already available for QCon attendees, and will later be available for readers of InfoQ. He has also published the slides of the presentation.

Rate this Article

Adoption Stage

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread


Login to InfoQ to interact with what matters most to you.

Recover your password...


Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.


More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.


Stay up-to-date

Set up your notifications and don't miss out on content that matters to you