BT

Your opinion matters! Please fill in the InfoQ Survey!

How and Why Etsy Moved to an API-First Architecture

| by Thomas Betts Follow 5 Followers on Aug 02, 2016. Estimated reading time: 2 minutes |

At QCon New York 2016, Etsy software engineer Stefanie Schirmer told how her company successfully transitioned to an API-first architecture that supports multiple devices, addresses server-side performance problems, and was quickly adopted by development teams.


The engineering team at Etsy has established its reputation with architecture that is optimized for change, facilitates continuous experimentation and allows them to deploy 50 times a day, so it might be surprising to learn they were dealing with serious performance problems a few years ago.


With a goal of 1,000 ms time to glass, their team needed to decrease server processing time of each client request. Unfortunately, the single-threaded world of PHP did not easily allow concurrent API calls, resulting in slow, sequential execution. Schirmer and her colleagues needed to figure out how to allow concurrency or they'd risk permanent performance problems. Complicating the matter, it wasn’t clear if the root cause of the performance degradation was backend issues or the nature of client requests.


Unfortunately, the development team couldn’t solely work on improving performance. In addition to supporting and upgrading the Etsy.com website, new features for moblie apps required increased extensibility from the platform. The solution to both problems required the API team to embrace a new design philosophy, while keeping the API familiar and accessible to the development teams.


Instead of a relying on a single API with a flat set of endpoints, Etsy created a two-layer API using meta-endpoints. Similar to a pattern used by Netflix and eBay’s ql.io, each of Etsy’s meta-endpoints aggregates several other endpoints. This enables server-side composition of low-level, general purpose resources into device- or view-specific resources.


The full stack forms a multi-level tree, seen in Schirmer’s slide below. The consumer-facing bespoke homepage is tailored to a specific view. It consumes a layer of concurrent meta-endpoints, which in turn call the atomic component endpoints. Only the lowest level components, which are not meta-endpoints, are able to talk to the database.

The layer of meta-endpoints reduced the complexity for composing a bespoke view, for both the website and mobile apps. However, single-threaded processing of multiple meta-endpoints would not meet the performance requirements. Etsy engineers utilized cURL to make parallel HTTP calls, going so far as to patch libcurl to meet their needs. Custom monitoring tools visualize the call hierarchy of a request as it fans out through the framework, allowing trouble spots to be located.

Additional internal tools were created to streamline adoption of the new API. Etsy’s first automatically generated API clients, which knew the exact parameters for each endpoint, accompanied by documentation, simplified the learning curve for developers. Rather than a single, one-size-fits-all approach to training, teams participated in pilot groups, code labs, lunch and learn workshops, and pairing with experienced developers.

Schirmer believes the story she told is a case study about architectural changes, and is transferable to other systems. The effort that was put into ancillary tools and services around the API certainly helped the platform team “sell” the new API to developers. In response, Schirmer and her colleagues are keeping the conversation with development teams ongoing, to ensure the framework will continue to evolve for everyone’s benefit.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT