BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Netflix Conductor, an Orchestration Engine for Microservices

| by Abel Avram Follow 4 Followers on Dec 20, 2016. Estimated reading time: 1 minute |

Netflix has developed an orchestration engine called “Conductor”, and has used it internally in production for the last year . During this time they executed some 2.6 million process workflows, starting with linear ones and ending with dynamic ones running over multiple days. Now they have open sourced Conductor, making it available to all those interested in workflow orchestration.

According to Netflix’ developer documentation on Conductor, the main characteristics of this engine are:

  • The ability to create complex workflows
  • The tasks are executed by microservices
  • The workflow blueprint is written in a JSON DSL
  • The engine provides visibility and traceability into process execution
  • The ability to pause, resume, restart, stop tasks
  • The task execution is generally asynchronous but it can be enforced to run synchronously
  • The ability to scale to millions of process flows

The Conductor’s architectural diagram is shown below:

conductor-architecture

Both the API and storage layers are pluggable, allowing one to use different queue and storage engines. Netflix uses Dynomite for storage but one can switch to another solution as long as an interface is implemented.

Workflow tasks are of two types: Worker, which are user tasks executed on a remote machine, and System, executed on the engine’s JVM. The latter are meant to branch, fork, and join the execution of worker tasks. Worker tasks communicate with the Conductor via HTTP or gRPC on HTTP/2.

Netflix has decided to create an orchestration rather than a choreography engine because the latter did not scale well in their tests. The issues mentioned were:

  • Process flows are “embedded” within the code of multiple application
  • Often, there is tight coupling and assumptions around input/output, SLAs, etc., making it harder to adapt to changing needs
  • Almost no way to systematically answer “how much are we done with process X”?

Conductor has been open sourced under an Apache License 2.0 on GitHub.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Looks like BPEL by Rafael Rocha

Anybody knows what is the main differences between BPEL pattern and the purpose of this framework ?

Re: Looks like BPEL by Matt F

It does look like BPMN to me. My hope is that that this does the same thing but in a more natural and easier way.

Re: Looks like BPEL by Jingfeng Zhao

BPEL is XML based. Netflix Conductor works with JSON.

BPEL in Microservices space... by chandra ganapathy

Purists say orchestration of Microservices for realizing business process are antipatterns as it increases dependency and instead lean towards choreographic interactions. I have always felt not all interactions can be made choreographic due to Eventual consistency - don't know when to state if a logical transaction has failed or still in process. It is good that I am not alone.

This is netflix re-inventing BPEL in Microservices space using its custom DSL.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

4 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT