Key Takeaways
- You can't buy ownership. Insourcing is imperative for modernizing legacy systems.
- Complex problems are cross-functional problems. Cross-functional problems are best solved in cross-functional teams.
- We have succeeded to achieve alignment with autonomous teams by using a mix of techniques that stimulate discussion and transparency, but also give a clear direction and guidance.
- An internal technology radar and a weekly town hall meeting on a particular subject are techniques that stimulate discussion and aid learning between teams, and as a consequence of that learning - alignment between teams.
- These techniques work well when combined with a strategic opinionated, carefully communicated technical direction that aligns on values, and a well developed ability to operationalize good practices in platforms that align on practice.
In this article we will describe how NAV (Norwegian Labor and Welfare Administration), Norway's largest bureaucracy, has achieved alignment in over 100 autonomous teams. We will talk about the history of NAV, and how we started this process with visionary leadership and insourcing. Then we will show you the techniques we use to align teams with respect to technology: two descriptive techniques - the technology radar and the weekly deep dive, and two normative techniques - the technical direction and internal platforms.
Six years ago at the Norwegian Labor and Welfare Administration…
NAV didn’t always have a DevOps culture. Six years ago, NAV didn’t have any in-house developers at all. We had no cross-functional product teams and did four huge coordinated releases four times a year. Six years ago, we were totally dependent on large consultancies. Our strategy was basically to not understand development. We valued project management, high level architecture, manual testing and operations. We were optimizing for handovers, deep in the plan-build-run camp, with large, long-running projects.
Working towards internal teams
In 2016, NAV got a new CTO, Torbjørn Larsen. He insisted that in order to succeed, we needed to reclaim the technical ownership of our systems. As a direct consequence, we started to insource. In six years we have grown the number of in-house developers from 0 to over 400.
That’s the elevator pitch. And while that pitch is true, there is a longer component to this answer. When Torbjørn joined NAV, he articulated an extremely appealing vision for what NAV could be, and his transformative leadership created a room for building a developer community necessary for realizing that vision - but building that community has been at least as much bottom-up as top-bottom. Even before 2016, there was this small team in NAV that built an application platform. Truls Jørgensen worked on that team (as a consultant). Looking back, that platform was pretty good for its time. It demanded a clear separation of application config and environment config, and offered automatic provisioning, zero downtime deploy, horizontal scaling, monitoring and logging in return. The problem, however, was that the rest of the organization did not actually use it. That caused us to do an assessment of the continuous delivery maturity model from the Continuous Delivery book. We found that while we did quite good on build & deploy, the organization was embarrassingly low in all other categories. What good is there in offering deployment in minutes when it takes months to change a comma in a simple web application due to incredibly inefficient organizational processes? That assessment formed much of Jørgensen’ motivation to start in NAV as the first developer in 2016. At first, it was really hard getting people on board, but then we got the ball rolling. Audun Fauchald Strand joined one year later, and working together they greatly accelerated the buildup.
Looking back, we can say that the insourcing has been successful. The cost is lower, because we have replaced external consultants with internal developers. We have delivered lots of value, both on the infrastructure side, speeding up, and moving to the cloud, but we are also far along the way of building the organizational units that frame the problem we need to solve, for many years to come. We have shown that we can have internal teams that build these new systems. Of course, there is a lot of learning going on. Going from projects to continuous product based development takes a lot of change, both in terms of culture, and in terms of governance. We try to build a governance structure that works with long-living teams. And that is also time consuming and difficult. The rest of this article describes the governance structures that we have put in place at NAV.
Aligning teams within NAV
Our internal technology radar is perhaps the most basic technique to align teams. It exists so that our 100 teams can share their technology choices. What have they tried, and what have they learned? The radar has the traditional Thoughtworks radar categories - “Assess”, “Trial”, “Adopt” and “Hold”. In addition, we have added another category - called “Disputed”. That's where the Spring framework and other such eternal arguments live. The radar itself is implemented with a simple Slack channel where suggestions and discussion take place - and a web app to show all the data.
Our technology radar is sometimes misunderstood as a list of allowed technologies. That is not what a radar is. A radar is descriptive, not normative. We think of it as a curated list of the technologies used. We have a small group of editors who collect the suggestions, and moderate the discussions. Our radar shows the breadth of our technology landscape. It gives our teams insight into what technology other teams are using, and that itself creates alignment.
We tend to summarize the philosophy behind the radar as “When it comes to technology, everything is allowed, as long as you broadcast it to all the teams”. That philosophy has proved to yield another cool consequence: it has effectively eliminated shadow IT. It is straightforward to see why - when it is safe to broadcast your technology choices, there's no need to keep them in the shadows.
In addition to the radar, we have the weekly deep dive on a particular subject. We have been doing this for six years, since we were around 10 developers. Although the scale has changed quite a bit- the weekly deep dive now attracts more than 400 people every week, the recipe is still the same.
One team takes the microphone, and presents something they are proud of, something they have learned, or something they are interested in to all the other teams.
We have learned that such an arena is a really efficient way of spreading good ideas across teams. This technique has a fantastic high return on investment. The investment is a dedicated, motivated group of people that manages to arrange this demo every week. That group is in return doing an incredibly important job - they are really changing the culture towards more openness, more kindness, more curiosity, and more learning, each week.
That is in addition to creating alignment on good ideas, aligning our way of working, sparking collaboration. As with the technology radar, it is a strictly descriptive technique.
The first normative part is the presentation laying out NAV’s technical direction. This is a presentation created by the principal engineers, and is an opinionated list of changes that we think that NAV should do. Our technical direction creates alignment by approaching each team with respect for the team autonomy. We are open about the fact that we want to challenge them to change something they are currently doing, But that change must and should be adapted to their context and abilities, and we also give them helpful guidance along their way. Also, we communicate that we understand that the team needs to balance different voices, not just ours, so we try to not push more on their plate than they can digest.
The most efficient way of getting alignment is achieved in a common platform, such as an application platform, a design system platform or a data platform. All of these platforms have one thing in common. They strive to make it easy to do the right thing. We try to avoid using a stick, and focus on building an incredible carrot instead. By doing this we hope to find the right balance between the innovative creativity of autonomous teams, and the large-scale efficiency of alignment.
A part of building this incredible carrot is Cake-Driven Development. We wanted to replace the question “Do I want to rewrite my old jboss application to a 12-factor stateless application running in docker on kubernetes?” with the question “Do I want cake?”
We went out to all the teams and said that every team will get cake, proper, delicious chocolate cake, when they get their first app running in production. And, of course, we weren't that sinister. We made all the teams that got cake take a picture, and we posted that in our slack-channel. Then the other teams saw that more and more teams migrated, and they wanted to be a part of the cool gang. They knew that they could ask other teams for help. We had built a community.. We soon saw that most of the support in our application platform slack channel was answered by other users, not by the team making the platform. And that is worth quite a lot of cake, to be honest!
How alignment enables autonomy
Alignment without autonomy is autocracy, and autonomy without alignment is anarchy. While it is easy to agree that aligned autonomy is desirable, actually getting to that state takes a lot of work. A common strategy for achieving alignment is to articulate a goal, and informing about that goal top down, and then expect that the recipients of that message will immediately execute on that goal. That is really an autocratic leadership mindset for creating alignment, addicted to command and control. We have seen this strong indication of sloppy leadership repeating at several organizations.
Alignment in the age of autonomy needs another approach. The goals you set must be understood and to some degree also welcomed by the teams, and that is your responsibility as a leader. The idea is to make the teams want to do what you want them to do. When the goal is understood by everyone, alignment is pretty easy.
When the pandemic hit us back in March 2020, we had to deal with a big emergency with hundreds and thousands of people being furloughed. The system handling the benefit for furlough is one of our old monoliths, and manual case-workers. The estimate was that there would be many months before all the applications would be handled. So we had to build a new system, create a shortcut, paying an advance of that benefit.
We built that system - from nothing to payout - in three days. We were mostly reusing components from other systems. And that was possible for us to do because our components are loosely coupled, mostly using Kafka for communications. So building this new system was mostly creating a frontend, and integrating the other components. And perhaps, most important: everyone was on the same page, perfectly aligned - from the prime minister to the developers on that team.
But that is not the usual way of working, of course. Creating alignment is a lot of hard, but necessary work in order to set the teams up for success. When successful, the teams are responsible for a problem and the solution(s) to that problem, within a bounded context. That means that issues outside of that context should be handled outside of that team, and that enables autonomy within their bounded context.
We also believe in standardizing communication patterns. We say that teams should communicate with other teams using asynchronous messaging, preferably using streams. This makes them loosely coupled, and when the teams have fewer dependencies on other teams, it is easier for them to make their own decisions.
The benefits of cross-functional teams
Most of the problems NAV aims to solve are cross-functional by nature, and cross-functional problems are best solved in cross-functional teams. Most of our problems have no perfect solution; they exist in the complex domain in the Cynefin framework. where we must balance different, cross-functional perspectives constantly in order to tune our solution to be as good as possible. We should not only set the teams up to be cross-functional, but there is also an important point on diversity here. As we create services for a very heterogeneous user group, that diversity should also be represented in the organization behind those systems.
When different people with different values and different professions work together, they tend to not only balance their perspectives, but also have a combined skill set that allows them to uncover deep organizational secrets that one profession alone wouldn't be able to. We need to aim for the best balance between different perspectives:
- The IT perspective is typically that the team should focus on building sustainable solutions. Because we are modeling a welfare state that is constantly changing, our systems should be adaptive to change for many years to come. That makes sense, right?
- From a legal point of view, the team should optimize for compliance to laws and regulations. That makes sense too!
- From a service design perspective, the team should focus on making sure that they provide a user friendly service. Wait, that makes sense as well!
- The finance perspective wants the team to focus on the financial side. How is the actual cost compared to the projected cost? The team should regularly evaluate if it is still worthwhile to continue solving this problem.
We need to balance all of these perspectives in order to be successful. We also see that innovation happens when there is cooperation between those perspectives. For instance, we have a system that automates the checking of all the conditions necessary to get a specific benefit. When we do this with computers, we can store a lot more data about the process. This data can be really useful:
- What conditions correlate? Can we simplify the law?
- Are there conditions that are irrelevant?
- Can we simulate what happens if we add or remove conditions?
Facilitating change within teams
We have a very heterogeneous set of teams, and a team's ability to change differs a lot. It depends on not only the level of internal technical ownership, but also on how adaptive to change their software systems are. It could be COBOL, running on a mainframe for the last 40 years. It could be a new shiny Kotlin service, running in docker in Kubernetes in Google cloud - and everything in between. With over 100 teams, our teams and systems have different starting points and different abilities to change, but we encourage every team to try to move in a direction. We do that by approaching each team with respect for their autonomy. We spend time with each team on our technical direction, and try to influence them to pursue a change. One such change is moving to the cloud. We spend time with teams that are hesitating, explaining the bigger picture motivation for the migration and showing the superior tooling that will be available to them. We make sure that questions they have get answered, and as they start their journey, we provide guidance along the way. If end of support for an on-premise infrastructure is approaching, and a team shows little interest in moving away from that, we push just a little bit, but we don’t really do much of that. For the important stuff, they need to pull.
Architecture is also important. Loose coupling, and the ability to change without the change affecting other themes, is really helpful. Also, they need the necessary quality to be able to change their systems. To achieve this, they need to continuously work on technical debt. Technical debt can arise as a function of time. We often explain this by asking the team to imagine a perfect software system that maps every single business requirement perfectly, and where all of the code is written so beautifully that even the most grumpy programmer gets tears of joy; even that system needs continuous love and care. Tomorrow, the business requirements can change. A pandemic could happen. Log4Shell can happen. A perfect code base today can have technical debt tomorrow.
What we learned
It takes time and effort to make teams go in the same direction. Balancing the need for autonomy, with the need to efficiently scale up, is difficult. We have also seen that other lines of work have started to work in the same manner that we are talking about. We now have a compliance catalog, that just like our technology-radar enables transparency. The compliance catalog makes it easy for a team to share how they solved compliance requirements. Other teams then use that knowledge for inspiration to carry out their own compliance assessment, such as how a team is handling personal data, what data is gathered from external sources, which systems are master of which data, and so on.
It all boils down to people. There has been this segregation between business and IT for ages, that has damaged how people on both sides think of each other. IT cannot be seen as a support function for business. In order to succeed with the digitalization at NAV, IT had to be integrated with the business once again - like it used to be in the mainframe days we’ve been told. The DevOps culture started as an IT thing, and now, when trying to integrate the business with IT, it is the same pain points that needs to be addressed. People need to be nice to each other, show respect for the depth of other professions and understand that the outcome of working together is greater than the sum of its parts.