Spotify's infrastructure team has shared how by prioritising data, they've built an automated data collection platform that enables data-driven decision-making in DevOps and has improved their developer productivity and product value.
The Spotify infrastructure teams use Gradle (Gradle Enterprise Edition) as a build system for their Android apps. It generates, collects and stores the data needed to understand their software in the context of the local development experience. It requires joint focus on data pipelines and visualisation on dashboards. There wasn't an established solution for data generation, collection and storage for iOS, so the team built the tools themselves.
Spotify has been investing in data for some time. The Spotify Tech Learning team has launched Data University, a series of training sessions that covered different aspects of data science and engineering, designed to help engineers solve product-related problems.
The Android infrastructure team brought those lessons to bear on their build times and their local development experience, but discovered that they lacked the data to drive decision-making.
Spotify has tackled this data demand by dedicating certain groups it calls tribes to providing the data infrastructure, and equipping the engineers with the building blocks to collect the data and feed it into visualisations. They indicate that plenty of challenges remain, for example to bring this data-driven approach to their architectural decisions.
The team used this new data infrastructure to shed light on where the technical and product teams should invest in to reduce build time. When they looked at the build-time trend, and the overall numbers of components in use across Swift and ObjC, it made sense to invest in Swift optimisation.
This technical investment into data-driven decision-making stands out against recent research by Harvard Business Review Analytic Services, which shows that only 7% of organisations give their teams the analytics tooling and resources needed to drive data-based decision-making and autonomy.
In essence, Spotify's approach is simple: the team produces questions they don't have the answer to, and then prioritises the work to answer them in the backlog. After the data is available and the question answered, the team collects feedback in an evaluation phase to understand if the work had an impact on the local development process. To prevent deterioration of the data quality, the teams have to build in quality checks on data consistency and data pipelines per component.
In the planning phase, the team uses historic data to identify scenarios for improvement. The data may not describe the current scenario, but it offers a baseline to identify improvements. If they already understand the build time for the system in a specific scenario, they expect to either keep the same numbers or improve them regardless of the growth of the codebase. This is critical because as systems become more complex, DevOps workflows generally become complex and opaque.
Agile naturally leans towards prioritising product, so the challenge in DevOps is to throw light onto the compromise between investing in features to increase product effectiveness or increasing development efficiency or improving service reliability.
During planning, the team introduces tasks that collect and display the data needed to validate their changes. The questions raised in this phase are one of the key outputs, e.g.: "Are we collecting enough information to check whether developers have the remote cache on?" or "How many components do they change on average in a single PR?"
As the infrastructure team's data initiative was recognised more widely internally, other teams began prioritising work related to the platform. Product teams started to value the data visualisations for validating the product discussions that drive the decision-making process for mobile DevOps teams.
Data-driven decision-making in product teams has helped to evaluate the effectiveness of solutions, and adoption satisfaction. Product managers often use user research from the very early stages to evaluate products. By contrast, the data-driven process brings this evaluation to product conceptualisation.
InfoQ's Data-Driven Decision-Making Series provides an overview of how the three main activities in the software delivery - product management, development and operations - can be supported by data-driven decision-making.