Tapabrata (Topo) Pal, VP of Enterprise architecture, and co-author of "Investments Limited", presented how Fidelity improved their DevOps strategy by investing in Inner Source, Open Source & Engineering Excellence.
After having spent 10 years at Capital One, Topo joined Fidelity in March 2021, mainly because he believed in their company's Culture Code focused on "happy and engaged people delivering customer value". In addition, Fidelity Culture Code states:
We believe in leadership at all levels and that titles don’t determine what you can and cannot do.
No roles, only role models.
At Fidelity, Topo took on the responsibility of their DevOps domain architecture and their open source program office. He saw that the company had been successful in implementing a multi-public cloud and hybrid strategy. From an agile and DevOps perspective, Fidelity deployed their first application on the public cloud in 2016. In 2017 the organization kicked off an organizational agility initiative, which enabled 50% of its applications to be on the cloud by 2022. With 18,000 developers across 1,600 squads, Fidelity has the goal to move 70% of their applications to the cloud by 2024.
To do this, soon after he joined the company, Topo conducted a listening tour, where he interviewed several leaders across the organization. To every leader he met, he asked the following three questions:
- What is going well today?
- What is not going well today?
- And what can we do to make it better?
Topo then sorted out and analyzed the challenges he heard, and identified four key opportunity areas:
Developer Experience:
- Onboarding is a challenge due to the volume of their processes
- Documentation: they had a lot of documentation, more than they’d ever need
- Context switching is exacerbated with the above and the number of tools available
The Tool Sprawl:
- Too many tools, technologies, and platforms, which is according to Topo not just a challenge at Fidelity, but an industry problem
- They have more tools that they can consume, leading to too many abstraction layers, frameworks and complications when upgrading
Security, Audit Compliance:
- Shifted right leading to late feedback loops
- Time-consuming audit
- Ownership: the main difficulty according to Topo. To define the future state, Topo asked: "Can the developers own the security and availability themselves, and can they decide what to do with them? Can they get feedback early enough in the development lifecycle?"
Metrics:
- Lots of metrics. Here as well, Topo mentioned that this is not just a challenge at Fidelity, but a problem that affects the industry
- Lots of different techniques used to measure
- And too many maturity models
In terms of metrics, Topo mentioned:
"We are comparing apples to oranges, and between apples and oranges it's very difficult to tell where the team is in their DevOps journey."
With these findings, Topo met with Fidelity's CIO, and together they agreed to take a collaborative approach, starting by getting everyone together to discuss and design the next steps. They formed the Fidelity DevOps Council, including participants from all functions across the organization. They then formed small groups, each tasked with identifying solutions to these challenges. They reported back to the council, presented their proposed solutions, and discussed the next steps.
The DevOps Council decided to invest in engineering excellence and focused its efforts in the following areas:
- A unified developer experience
- Tools standardization
- Continuous compliance
- Contextual metrics
By "contextual metrics," the DevOps Council means "measure what matters to us, that matters to a particular set of developers, a particular set of products, business and so on." For each metric proposed, they would answer why, who, when, and how? They were cautious about the "observer effect": they didn't measure things that might have affected the behaviors of the people; instead, they promoted "measuring to improve ''. Topo proposed that every request to measure something should be stated in the following format:
"As a (role), I want to measure (this process), so that (benefit and improvement)"
And he shared the following example:
"As a product owner, I want to measure Team's Production Access Frequency, so that I can understand Team's Technical Maturity and help improve it."
To support this effort without having to stand up a shared platform, the Fidelity DevOps Council invested in inner sourcing and Fidelity brought in open source to scale up. They developed an open-source framework focused on "Incubate - Fund - Govern - Manage - Promote - Operate". In 2021, they kicked off five new projects, among which one of them produced to date 174 features, and 3500 commits, supporting 12 business units, with 50 active contributors.