Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News InnerSource: Internal Open Source at PayPal

InnerSource: Internal Open Source at PayPal

InnerSource is the name given to a development approach that applies open source software practices to the way organizations' develop software internally. Cedric Williams, technology leader at PayPal, explained at OSCON how PayPal is experimenting with InnerSource to break down silos, grow collaboration and increase productivity.

The journey started around 18 months ago. The Checkout Platform team was spending 2/3 of their time rewriting code submitted by the regional teams. The regional teams, among other responsibilities, ensure that PayPal complies with the different regulations of the many different countries it works in. This state of affairs was not good for anyone. The Checkout Platform team was not doing productive work. The regional teams were spending time writing code that would be thrown away.

PayPal drew inspiration from open source software practices, in particular from the Apache Software Foundation. They found that the organizing principle for each project is a layered pyramid: users, on the bottom; contributors; trusted committers; and lead architects\developers on the top. PayPal reckoned that in its case the most important changed was to introduce the "trusted committer" role.

Finding the right trusted committer was not an easy decision: only 10% of the Checkout Platform team has that role. Answering a question from the audience, on the technical skills required to be a trusted committer, Williams said that they must have deep technical skills and know the codebase by heart, but they don't have to be the best. Trusted committers must also have the right people skills. They need to be coaches. They need to communicate in a positive and clear way. For instance, instead of saying that "this code is unacceptable", they need to say "here's what I need you to do so that I can accept your code. Here's why: (...)".

As expected, introducing trusted committers was fraught with political risk and had to be handled with care, both outside and inside the Checkout Platform team. To make the whole change easier to accept, trusted committers not only reviewed the regional teams code submissions, but also the ones from the rest of the Checkout Platform team.

The results were visible after 6 months. The Checkout Platform team spends 0% of its time rewriting code and just 10% reviewing submissions. The team was able to do a major refactoring and a 4x increase in performance without planning for it. The mindset moved from blocking change to mentoring and coaching. One important byproduct was that all relevant communication was done in writing, mostly on GitHub: PayPal uses GitHub for their own OSS and GitHub Enterprise for closed source. This allowed the knowledge to be shared and spread across all the teams.

Before InnerSource, PayPal tried different approaches: mandating change from the top and colocation.

The first approach is the traditional one. Define strict rules and stakeholders, mandated from the top, to make people accountable. This didn't solve the problem, but it did generate a lot of meetings.

Colocation worked a bit better. The senior engineers from each of the regional teams were brought to San Jose to work side-by-side with the development team. Colocation led to knowledge sharing and a better understanding of each team's motivations. Unfortunately, once the engineers returned to their regional teams, they became the bottlenecks. They were unable to find the time, or didn't have the required skills to pass on what they've learned.

Williams offered some advice on getting started with InnerSource. As a first step, restrict your trusted committers to about 10% of your engineers, require all code reviews to be publicly available and ensure those code reviews focus on what can be done to make the code better. Second, require mature and respectful behaviour on all conversations. Third, share your experiences. PayPal will host the first InnerSource Summit on November 9th, at San Jose, California. PayPal also sponsored a short book on the subject, for those who want to learn more.

Rate this Article