BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Taming Dependency Hell within Microservices with Michael Bryzek

| by Daniel Bryant on Jun 13, 2015. Estimated reading time: 3 minutes | NOTICE: The next QCon is in New York Jun 26-30, 2017. Join us!

Michael Bryzek, co-founder and ex-CTO at Gilt, discussed at QCon New York how ‘dependency hell’ could impact the delivery and maintenance of microservice platforms. Bryzek suggested that dependency hell may be mitigated by making API design ‘first class’, ensuring backward and forward compatibility, providing accurate documentation, and generating client libraries.

Quoting the Wikipedia entry for dependency hell, Bryzek introduced the phrase to the audience and suggested that almost every developer has witnessed this at some point in their career:

Dependency hell is a colloquial term for the frustration of some software users who have installed software packages which have dependencies on specific versions of other software packages

Bryzek discussed how Gilt.com, a luxury brand flash sales ecommerce site, moved from a traditional Ruby on Rails monolithic stack to a Scala-based microservice platform over the course of five years. This, combined with the growth of the technical team to over 150 people, lead the team to ask how dependencies should be managed, specifically dependencies that were used in libraries.

Four strategies for minimising the pain associated with dependency management were presented:

  • API design must be first class
  • Backward and forward compatibility
  • Accurate documentation
  • Generated client libraries

Bryzek proposed that several of these strategies can be implemented with appropriate tooling, and accordingly he has created a Scala-based tool named apidoc for this purpose. apidoc has no external dependencies on an existing software process nor runtime, and is open source and provided as a free to use software as a service (SaaS). Bryzek commented that over 100 services have already been built at Gilt with apidoc.

Diving into each of the strategies mentioned above, Bryzek proposed that the design of an API and the associated data structures are often the hardest things to change within a system. Therefore, they should be designed up-front, and the corresponding artefacts integrated into the design process to ensure accuracy between a service and associated clients. This is in essence, “schema first design”.

In regards to ensuring backwards compatibility of an API or client library, Bryzek suggested that developers should imagine processing historical records whenever a change to an API is made. Fields added to an API should either be optional or have sensible defaults, and renaming of fields should never occur - instead new models should be introduced, and a migration path be provided where appropriate.

Forwards compatibility is also an important concern, and for this scenario developers should imagine processing new messages arriving to the service with new data. The benefits of Postel’s law, ‘be conservative in what you do, be liberal in what you accept from others’ are relevant to this discussion. APIs (and corresponding client libraries) should be versioned to aid developers, and one such technique for versioning is semantic versioning.

Additional considerations for establishing forwards compatibility include being careful when using enums, for example when using binary interface definition languages such as Apache Avro or Apache Thrift, and being careful when throwing errors if an unknown field is encountered within an API’s data structure. Ideally tooling should clearly indicate when things have changed within an API or client library, and the apidoc tool provides this functionality.

Accurate documentation for an API is also essential, and this may be automatically generated, for example using a tool such as Swagger 2.0. Bryzek also discussed that although he was initially sceptical, he now believes that generating client libraries automatically from an API can be beneficial. For example, boilerplate API consumption code can be removed (or encapsulated within the client library), and consistent naming for fields and actions can be maintained across services. However, care should be taken to ensure that the client library has minimal external dependencies (so that the dependency hell is not pushed down into the stack), and generating a client that developers will enjoy using can be very challenging.

Additional details about Michael Bryzek’s “Microservices and the art of taming the Dependency Hell Monster” talk can be found on the QCon New York 2015 website, and more information about the apidoc tool can be found on the project’s Github website and the Gilt Tech blog.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Not tied to microservice by Erik Gollot

This problem is not linked to microservice architecture. We have the same problem with monolithic apps. Microservices 'vendors' often hide or omit it most of the time.
The problem is just how to manage interface evolution and it's not a technical topic, it's a business one. You need to deal with the semantic and business rules. And more than an api problem, it becomes quickly a database schema problem, which default value, is the default compatible with new business rules ?...
So it's not new, it's just that in all SOA like discussion we often forget to talk about data and semantic

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

1 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and dont miss out on content that matters to you

BT