Spotify runs more than 250 online experiments annually on its Spotify Home platform, which are used by dozen of different teams. To accomplish running experiments at such scale, Spotify uses a number of different tools, explains Spotify product manager Nik Goyle.
At Spotify, our Home serves as a personalized surface where users retrieve familiar content and discover new content tailored to their preferences. [...] Home experiments focus on pushing the boundaries of personalization, exploring innovative ways to tailor content and programming strategies.
At Spotify's scale, it makes sense to develop custom tools to improve the efficiency of online experiments, especially A/B testing. One such tool is dubbed Home Config and is aimed at allowing developers and testers to fine-tune personalization strategies. Goyle describes it like a configuration-as-service tool, making it simple for technical as well as non-tech users to create personalized experiences to test out:
[Home Config] allows them to define parameters related to ranking, content, visual treatments, and more, ensuring a personalized and tailored experience for users.
Another essential component of Spotify's solution is the Experimentation Platform, which enables releasing configurations created using Home Config into production and speeding-up the experimentation process.
It provides experimenters with a comprehensive interface to design, launch, and monitor experiments. With EP, experimenters can define experiment parameters, set up control and treatment groups, track metrics, and analyze results.
A third tool, Home QA, is a front-end application able to simulate Home requests to ensure every experiment does not break anything before launch.
Another critical dimension to Spotify approach to experiments is coordination. This is backed by two additional tools, one Experiment Tracker used to prioritize and monitor all experiments from a centralized location, and the Experiment Validation Assistant, which validates proposed A/B tests by checking they are not misconfigured and providing actionable data in an automated way.
EVA decreases overhead and ensures experiments adhere to predefined criteria. The results of EVA’s validations are shared in a designated Slack channel, enabling swift feedback and necessary adjustments.
While not all organization will have the resources to implement their own experimentation tools, Spotify's approach illustrates important requirements for a successful strategy to carry through experiments at scale, including how to configure experiments, deploy and run them, ensure quality is preserved across them, and how to streamline communication. If you are interested in the full details, do not miss the original article.