Maria Jernström and Jason Palmer, two product managers at Spotify, shared how the company enables their development teams to operate quickly and in alignment. The Platform Developer Experience tribe builds CI/CD tools, product creation tooling, and paved paths with a focus on automating common processes.
Spotify employs a DevOps model where each team owns and is responsible for their feature from end-to-end. While this allows teams to be highly autonomous, Jernström and Palmer note "it has also created fragmentation among our toolsets and engineering practices." Gary Niemen, product manager at Spotify, indicates this fragmentation leads to a form of "rumour-driven development" where "the only way to find out how to do something was to ask your colleague".
To help decrease the fragmentation, the Platform Developer Experience (PDX) tribe developed a set of tools and processes, such as the Golden Paths. Golden Paths present a set of best practices organized by type of engineering project, like client development, data science, or machine learning. As Niemen puts it, they provide "the 'opinionated and supported' path to 'build something'". These paths are hosted within Backstage, Spotify's developer portal.
Similar to the paved road concept popularized by Netflix, the Golden Path provides a standardized experience that scaffolds out the necessary project skeleton. Each path is accompanied by documentation that describes the best practices for this work. Tools that are on the path are accessible via Backstage.
The focus on consistency aligns with how Karl Stoney, delivery platform lead at Auto Trader UK, describes how Auto Trader enables their development teams. Stoney stresses that consistency in implementation is vital. This led their platform team to decide to abstract product teams from their platform by implementing a platform-as-a-service (PaaS) on top of tools like Kubernetes and Istio. This approach has led to easily pulling metrics and traces from all of their services in a consistent fashion. As Stoney notes, "consistent observability data enables you to easily build platform level tooling which benefits everyone. Automation can collate relevant information and reduce your MTR."
When creating the Golden Paths, Niemen stresses that they should have a clearly defined audience and one primary focus. Within Spotify they chose a new hire engineer as their target audience. As all new hires review the paths and tutorials as part of their onboarding, the team is able to gather plenty of feedback. To meet the needs of this audience the paths are step-by-step guides that contain all necessary steps to complete the path. As Niemen puts it, "the Golden Path is the opinionated and supported path to build your system and the Golden Path tutorial walks you through this path."
Since the path is meant to represent the entire workflow they found they commonly receive the feedback that the path documentation is too lengthy. However, Niemen encourages shortening the actual path, not the documentation. By automating portions of the process, the step-by-step documentation can be simplified while still maintaining the overall outcome of the path.
By first writing out the Golden Paths and then producing tooling and automation to shorten the paths, Spotify's platform teams ensure they are building components that will be beneficial. This aligns with advice that Camille Fournier, managing director, head of platform engineering at Two Sigma, shared in an InfoQ interview. Fournier stresses the importance of not investing in "software that is building to be built. When platform teams build to be building, especially when they have grand visions of complex end goals with few intermediary states, you end up with products that are confusing, overengineered, and far from beloved."
By optimizing a predefined path, teams can leverage the feedback from their users and automate the most time consuming parts first. This also focuses the platform team around a product they can deliver. Fournier notes that "great platform teams can tell a story about what they have built, what they are building, and why these products make the overall engineering team more effective."
Spotify still allows room for teams to stray from the path as needed. They empower teams to choose the tools and methods that fit with how they operate. To ensure they can still leverage the work that the PDX tribe has done, they provide options for leveraging their services. For a zero configuration experience, teams can use the templates directly from Backstage. For teams that need some amount of configuration, they support custom CI/CD pipelines. Finally, for teams that want to build on the platform but not leverage the pre-built templates, their CI/CD tool Tingle exposes APIs.
Additional information regarding the work that the Spotify PDX Tribe has done can be found over on the Spotify engineering blog.