Engineers at Uber created their own remote development environment to improve developer experience and productivity by fixing a number of issues brought about by their adoption of a code monorepo.
Uber's remote development environment (RDE), named Devpod, follows the same philosophy as other recently introduced RDEs such as GitHub Codespaces, JetBrains Space, and AWS CodeCatalyst. This includes the possibility of setting up a project in a few seconds, centralizing security, and improving performance thanks to the huge processing power available in the Cloud:
Moving from laptop to cloud enabled us to experiment with machines that have 90+ cores and hundreds of Gigabytes of RAM.
One of the driving factors behind Uber's decision to adopt a remote development environment was the inherent complexity of their huge monorepo, which consolidated over 4000 services, 500 Web apps and many tools across thousands of repositories.
Our move to monorepo provided several important benefits including better dependency management; consistent and universal library versions in production; single, centralized build platform (Bazel); easier support for a standard set of tools and processes; and improved code visibility, collaboration, and code sharing.
As Uber engineer Leo Horie explained on Hacker News, concrete examples of the advantages brought by the monorepo is fixing log4j vulnerabilities in a centralized way, simplifying npm auditing, and so on. He also stresses that the main problem a monorepo is trying to solve is not technological, rather it is a people problem:
For example, say you cron CI job fails. Then what? Someone needs to look at it. It's easier for someone to fix things they currently have context for (e.g. if I upgrade Python or update some security-related config and something breaks, I can reason it was my change that broke it), vs an unsuspecting contractor getting around to some backlog task 6 months after the fact with no context.
Unfortunately, the monorepo made builds slower, especially on laptop and when working remotely, with cloning and setting up a project taking hours.
With their current Devpod implementation, though, Uber engineer have been able to speed up a number of important metrics, including git status
and Go build times, by a factor ranging between 1.8 and 2.5. It's hard to say if this is a good result, considering the amount of effort that Uber must have put into Devops, but Devpod adoption inside of the organization at over 60% of the engineering workforce, seems to indicate this was the right move for Uber.
To create Devpod, Uber chose to go with Debian-based Docker images and Kubernetes as a deployment platform. Kubernetes clusters are deployed in several locations to be closer to engineers and reduce latency. To make management easier, they also created a centralized Web app and provided mechanisms to keep all environments up to date.
Uber engineers are still hard at work on Devpod to make it faster to clone a project, to introduce ephemeral environments, improve IDE experience, and allow for team-specific configuration.
It is not clear yet if Uber will ever open source Devpod, but their decision to migrate to a remote development environment could be a sign of a growing interest towards RDEs, as attested by Slack and GitHub cases, too.