BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Netflix Conductor, an Orchestration Engine for Microservices

Netflix Conductor, an Orchestration Engine for Microservices

This item in japanese

Bookmarks

Netflix has developed an orchestration engine called “Conductor”, and has used it internally in production for the last year . During this time they executed some 2.6 million process workflows, starting with linear ones and ending with dynamic ones running over multiple days. Now they have open sourced Conductor, making it available to all those interested in workflow orchestration.

According to Netflix’ developer documentation on Conductor, the main characteristics of this engine are:

  • The ability to create complex workflows
  • The tasks are executed by microservices
  • The workflow blueprint is written in a JSON DSL
  • The engine provides visibility and traceability into process execution
  • The ability to pause, resume, restart, stop tasks
  • The task execution is generally asynchronous but it can be enforced to run synchronously
  • The ability to scale to millions of process flows

The Conductor’s architectural diagram is shown below:

conductor-architecture

Both the API and storage layers are pluggable, allowing one to use different queue and storage engines. Netflix uses Dynomite for storage but one can switch to another solution as long as an interface is implemented.

Workflow tasks are of two types: Worker, which are user tasks executed on a remote machine, and System, executed on the engine’s JVM. The latter are meant to branch, fork, and join the execution of worker tasks. Worker tasks communicate with the Conductor via HTTP or gRPC on HTTP/2.

Netflix has decided to create an orchestration rather than a choreography engine because the latter did not scale well in their tests. The issues mentioned were:

  • Process flows are “embedded” within the code of multiple application
  • Often, there is tight coupling and assumptions around input/output, SLAs, etc., making it harder to adapt to changing needs
  • Almost no way to systematically answer “how much are we done with process X”?

Conductor has been open sourced under an Apache License 2.0 on GitHub.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Looks like BPEL

    by Rafael Rocha,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Anybody knows what is the main differences between BPEL pattern and the purpose of this framework ?

  • Re: Looks like BPEL

    by Matt Friedman,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    It does look like BPMN to me. My hope is that that this does the same thing but in a more natural and easier way.

  • Re: Looks like BPEL

    by Jingfeng Zhao,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    BPEL is XML based. Netflix Conductor works with JSON.

  • BPEL in Microservices space...

    by chandra ganapathy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Purists say orchestration of Microservices for realizing business process are antipatterns as it increases dependency and instead lean towards choreographic interactions. I have always felt not all interactions can be made choreographic due to Eventual consistency - don't know when to state if a logical transaction has failed or still in process. It is good that I am not alone.

    This is netflix re-inventing BPEL in Microservices space using its custom DSL.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT