In a rather detailed article, Slack engineers Ahmed Eid and Arpita Patel provided an interesting peek into the processes they adopted along the years to improve developer experience in a number of distinct areas and the tools they used to that aim.
Developer experience at Slack is catered for by a dedicated team of eight people, which formed to provide the answer to growing costs as the organization and its development team grew. Among the areas of the development process that caused the most cost, Slack engineers focused on merge conflicts, long-running CI jobs, flaky tests, and CI infrastructure failures.
While developers can learn to resolve some of these issues, the time spent and the cost incurred is not justifiable as the team grows. Having a dedicated team that can focus on these problem areas and identifying ways to make the developer teams more efficient will ensure that developers can maintain an intense product focus.
Estimated at $2,400,000 yearly cost per 100 developers, merge conflicts were the single most expensive pain point. This resulted out of Xcode project merge conflicts, concurrent merges to main, and longer pull request review process.
Xcode project are notoriously hard to merge without incurring in multiple conflicts. To address this problem, Slack engineers used Xcodegen to generate .xcodeproj files from a YAML file, a much more forbearing format.
Multiple concurrent merges to main bring an increased risk of merge conflicts, halting the merging of additional PRs until the conflict is resolved. For this, Slack adopted Aviator to queue all PRs and process them one by one. Instead of merging a PR into main directly, Aviator attempts to merge main into a developer branch. If that step breaks main, the PR is rejected and the author notified.
Finally, to speed up the pull request lifecycle, Slack engineers found it useful to introduce timed alerts for PR assignments, comments, approvals, and direct messages for successful builds including the option to merge the PR without leaving Slack. To accomplish all of this, they created their own GitHub bot, called MergeBot.
Mergebot has helped shorten the pull request review process and keep developers in flow. This is yet another example of how saving just five minutes of developer time saved ~$240,000 for a 100-developer team in a year.
Luckily, GitHub supports a similar feature, called scheduled reminder, albeit it does not provide one-click merge from the message itself.
Improving the PR/merge process was not the only action taken at Slack to improve dev experience. Another area which incurred high costs was testing and failures in their CI infrastructure. On the first count, the solution was parallel test execution along with a strategy to only run the tests strictly required for a given PR based on the PR diff. On the second count, BuildKite proved effective to increase CI-infrastructure reliability.
According to Slack, improving developer experience both made developers happier and reduced the overall development costs. If you are interested in the full details about how Slack achieved that, do not miss the original article.