BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Presentations Breaking Hierarchy - How Spotify Enables Engineer Decision Making

Breaking Hierarchy - How Spotify Enables Engineer Decision Making

Bookmarks
52:12

Summary

Kristian Lindwall shares how Spotify approaches the problem of fully leveraging the capacity of everyone in the organization. They have made efforts around how they position their engineers equal to managers, how they drive technical strategy and make decisions. He talks about useful models for structures and decision making, and how they approach leveraging the fully capacity of every individual.

Bio

Kristian Lindwall is an engineering leader at Spotify. He was managing parts of the Agile coach practice at Spotify for a few years and has been very active in supporting the growth of a strong agile and lean approach in the company. After that, he has managed the engineering team in San Francisco and most recently a 40 person team in New York.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Lindwall: I've been at Spotify for about six years now, and one of the most important problems I've been thinking about throughout my years there and before is this concept of how do we get to everybody's full capacity? How do we bring everybody's full capacity to the table and how do we break down the hierarchy? I'll get into that in a minute.

When I think about leadership and structures, I like to go to myself and look at what I do outside of work. One of the things that I'm really passionate about is music, that's always been the case. I play the drums, I've been in many bands and collecting music has been a big thing for me as well. When I was a kid, I had a lot of projects going on and by far my favorite band as a kid was Guns and Roses. Slash is, I think, one of the best guitarists. I had a project at the time where I wanted to collect all the world's music, but at the time there was no streaming. Cassette tapes were basically how we recorded.

What my friend and I did was, we started recording music on cassette tapes and tried to make the perfect mix, and we spent hours every day coming home after school, making these mixes. There was nobody telling us to do this. We were extremely passionate about this. I don't know how much time we spent, at some point we got to the point where we were, 'We don't know where this is heading.'' A few years later, you could write CD’s and fast forward another 20 years, and the world of music has changed completely, but the problem is still there. How do you create the perfect mix?

About 20 years later, there is a group of engineers sitting in New York in Spotify, thinking about exactly the same problem. Talking to them afterwards, I think they have the same level of passion that I had back as a kid to solve this problem. They were given a lot of space, a lot of autonomy, a lot of freedom to test different things. They had a lot of resources accessible to them. Eventually, they came up with this idea of a weekly mixed tape. We had all these recommendation engines, and the engineers and the team came up with idea that we could create a mix every week that we would release to our users, that would be new every Monday, which was the birth of Discover Weekly, which is a super successful feature with Spotify today.

The point of this whole story is, I think people are by nature very motivated to do good work. What tends to happen in organizations sometimes is that we constrain that energy and we shut them, down rather than enabling them, which is what I'm here to talk about. What happens in reality is that we create this feeling of you're having all of these ideas, but you're left out there in the desert and no one is hearing it. Then you hear somebody say something like "Our employees are our most valued resource." Who's heard that before? I think it's true, but sometimes we're not successful in showing that.

Where Does It Go Wrong?

Where does it actually go wrong? I think it's easy when we start. We want to start a business, there's usually one person, "Let's do this." You don't need to think too much about organizational structures, performance review programs and so on, you just do your thing. You hire another 20 or so, you find a product market fit then you maybe you launch a product. It's still quite easy, it's like this group sitting right here. You can all know each other's name and collaborate effectively, but then we're getting to the size of this room or maybe the double of that, and it starts getting a little bit trickier. Then we start thinking about how do we get all of these people to effectively collaborate. We need to start introducing some sort of structure, which is where we typically see some sort of hierarchy.

Not necessarily anything wrong with a hierarchy, I think most organizations have that. The problem tends to be how we implement it and that reverse gravity drives decisions up to the top and the further down you are, the less influence you have, and you start adding all sorts of structures to get people to do what you want them to do, and so on. This is what I want to talk about. How do we break that concept? I will group this into three areas where I quite often see problems. This is what I think is important.

The first one is, how do we break this problem of the hierarchy? One is to slice the pie and give people a piece of it. We need to make sure that we have, in the case of Spotify, several thousand people; it's a really big problem space. We need to slice that somehow and we need to give a group of people full ownership of a part of that problem.

The second area is that we have to stop telling people what to do. We can tell people what we need, what we desire, what problem we need people to solve, but stop telling people what to do.

The third one is, I think we need to become way more intentional about how we make and distribute decisions across the org. It's quite easy to push decisions up to the top of the pyramid, rather than effectively distributing it and have the people closest to the work and most equipped to make the decisions, actually make them. These are the three areas I'll go into a little bit more in detail.

Slice the Pie and Give People a Piece of It

Silicon Valley venture capitalist John Doerr put it really nicely, when he said that we need teams of missionaries and not teams of mercenaries, and I think this is a good one to think about. We want people who believe in a mission, that they come to work every day and want to solve a problem; not people who are coming there because they're paid to deliver on a task or chew off the backlog or however you look at it. How do we accomplish that? I think this is a huge deal at Spotify. We think a lot about how we set up teams and how we provide them with a scope, a mission that they can run with quite autonomously, so I’ll give you an example of what that looks like.

This is a team. I used to work in San Francisco where I headed up a team that we call App Integrations at Spotify. Their mission is to deepen relationships between artists and fans by extending Spotify into their favorite apps. What we did out there was we built our relationships to partner companies. We worked a lot with Facebook and Twitter and others. Wherever you saw Spotify in another app, that was what we did and the team out there. That was very clearly defined as a mission. We had a set of metrics that other people and we ourselves, could measure ourselves on, but we own that. We had all the people we needed in the team to solve for that mission and we could run with it. As long as we were moving our metrics, it was fine. Nobody was really asking too many questions.

We had a sibling team. In this case, they were based in Stockholm. They were called Home Consumer Electronics. Their mission was to fill homes with music that inspires every moment. They are working with speaker integrations and TVs and stuff like that. Then there were another few teams with similar missions. We all form what we call the Tribe at Spotify, in this case, the partners and platform experiences tribe, which spanned Stockholm and New York, that also has this mission of tailoring the Spotify experience to platforms and partners. The common denominator here is that all of us were working with platform and partners. We were extending Spotify out. We were not working with the main apps, but wherever you saw Spotify.

The work was quite similar nature, which is what made sense for us to be part of the company, but at the same time we could slice it even further and take our bite of it. App integrations was quite autonomous or independent from Home Consumer Electronics, for example.

Then we have a bunch of other tribes. To give you another example, the core experience tribe was defining and evolving Spotify's core experience, building essentially the main apps, the features that some of you might be using on your phone. All of us, in this case, the consumer experience mission had the mission of creating the most delightful music experience. There are a few hundred people who together build the Spotify main features that you're using, whether it's in the mobile app, on the desktop app, in a partner app. It still made sense for us to be grouped in some way. The whole point here is to show you an example of what it looks like to scale this mission. Each level here has a lot of space to deliver on the mission. I think this is really key when it comes to the first step to breaking the hierarchy. It’s about making sure that you carve out your space, because that forms the contract between the team and the rest of the company.

This is another view just showing a few tribes. I don't know how many we have today, we have quite a lot of tribes. One problem that we have been facing over the years - the flip side of a very high degree of autonomy is a lack of alignment sometimes; we want this autonomy to exist in the teams, but we also need teams to align somewhat because we're not completely independent. There are a few things that we have done over the years to counter that problem. One is guilds which is essentially a community of practice. If you're a mobile android engineer, you're part of an android guild and there' are ways for you to align with other android engineers across the company. We have internal conference and stuff like that.

Another thing that is a little bit more recent is that most of the tribes - and this has been both a bottom-up and also more lately a push is what we call a TSG or Technical Steering Group, you might have something like that, even called the same thing. The idea here is that in the tribe all the different teams have some representation in a group that can make more overarching technical decisions. In the case of the PPX tribe that I was talking to, let's say we want to have an overview of the full playback stack on all of these different clients and make some architectural decisions together that we want the whole tribe to align behind. That is typically a decision that would happen in this TSG group, which is a group of engineers who have also in themselves a lot of space to make decisions. One or two managers are typically part of these, but it's mainly engineer-driven.

Most of the tribes have one of these. Then we have mechanisms for them to align with each other, so we can drive a more overarching technical strategy for the company where there is also another group - this is going to be the last abbreviation, I promise - TAG, which stands for Technology Architecture Group. This group has representation from a lot of the tribes across the company and can drive strategy and direction for the whole technology organization.

The first problem, or first solution perhaps, to addressing the hierarchy problem is to slice the pie and build the organization around autonomous mini-startups and enable them to align effectively.

Don’t Tell the Organization What to Do; Tell Them What You Want

The second one here is to stop telling the organization what to do, and rather tell them what you want. To explain this a little bit further, I think a common problem for many organizations is that you have this plan that you expect people to follow and that you expect will lead to a certain result. Then the problem is that when the result is not what you wanted, you start looking at the actions and the plan and you start asking questions. You see that people weren't exactly following the plan, and then you start managing people more closely to follow the plan. You see that the results you were expecting are not the ones you got, so then you start questioning the actions of the plan and you start asking for more information perhaps, from this team to further understand why the plan didn't work.

Lastly, since the results at the end of the day are not what you anticipated, you start questioning the plan. This whole becomes a very vicious cycle of essentially micromanaging people to follow the plan, to give you more information so you can be able to plan better and then you invest more and more time into planning. I don't know if anybody has seen this before - a few people nodding, I think this is common. It's so hard to do this in the good way. What's a better way to flip this around? What we need to do is we need to start talking about the results that we want first. What is it that we desire? What is the intent here? What is the problem and why is it a problem? Then back to this concept of autonomy, we need to allow people and teams to define how to do it. Then also give them the freedom to adjust actions in line with this intent.

This sounds easy enough. How then do we actually do this? To look at that a little bit more closely, I wanted to start with a model of how people actually function. This is a great one. This is one representation of how people actually make decisions and get to conclusions. We all start with the world as it is, objective data. If you would record the world with a video camera, this is what happened, but then from there, we start picking pieces of data based off of our own biases and so on. Off that data, we add meanings based on our own experience, and then we make assumptions, we draw conclusions, we adopt beliefs, and then we take actions. That's the full range of how we get to a decision, which means that if we give two different people the same piece of information or the same task, they're going to do different things most likely.

What I think we need to get really good at here is to be able to express intent to give the problem in a way that taps into this whole model. This is where we came up with this model that we call DIBBs. That was another abbreviation, I'm sorry. What this is, is essentially a framework or a simple argumentation framework around this whole spectrum of thinking, where we start with the data, so objective points or sets of points that objectively describe the customer or the product or the world. This are pieces of information that you can't argue with, they're true. Then from this data we have an insight, so a learning or a theory or conclusion drawn from this data. From that we build up a hypothesis from these insights that offers direction to a team or a set of teams. Then based on that hypothesis, we come up with a bet or a set of bets which are decisions to test these hypotheses or beliefs. Then we put this in writing, and that is how we communicate this intent.

The good thing about this is that if you get this whole series of information, you can then challenge the data. "Is it incomplete? Why did you pick this data? I have different data, I have more data." You can challenge the insight. "I actually think that you made a giant leap here when you made that conclusion" and so on, which makes it way more easy to argue around this framework. To give you an example of this, if you go back a few years, Spotify did not have a free version on mobile. This is a quite simplified version of the DIBB at the time, but it looked something like this. We could see that mobile sales globally were increasing. We could see that on our own service, the mobile streaming was increasing relative to desktop. We could also see that our mobile engineering team was very small. This is again data points.

From that, we drew some conclusions. We had some insights here, which is that it seems mobile is going to overtake desktop. Our mobile team is also very small, it's disproportionate to our desktop team. Then a few beliefs here, there were way more in reality, but we need to become mobile first because if we look two years or so into the future, mobile will overtake desktops so we need to swap that paradigm, and to be competitive, we need a free tier mobile. Let's launch a new product, mobile free tier and let's ramp up the mobile engineering team to be able to meet this new paradigm. That's a simplified version but, that's how the framework works in practice.

What we usually do with these is that we put these into an RFC, Request for Comment, which is a common tool also. It's a document, a shared Google Doc is what we use most often in Spotify, it's very easy to share. Then we keep them open for a week or two or a few weeks. People can read this DIBB, this framework, and add comments and challenge add, "Let's add this piece of data, let's tweak it this way," and then after a few weeks we close it. The people who get invited to participate or comment depends on the type of problem that we're talking about. They're usually spread quite broadly.

The second principle here to think about is to express intent, not actions, and get really good about arguing what to do. Arguing what to do means you have to argue with the full range of arguments, from the data all the way to the thing that you want to do.

Be Intentional about How to Distribute Decisions

The third and final idea around how to break the hierarchy is to be super intentional about how to distribute and make decisions. I think that the most common way to think about an org, if you think about it as a geometrical shape, is to look at it as a triangle. It's where we started, the further up you are in the hierarchy, the more decision-making power you have, which I think is true for most companies. The problem though if you take this too far, is that the further down in this pyramid you are, the less of your capacity we actually leverage, which I don't think anybody would want if you were looking at it like that.

I think the mental model that I like and that we like at Spotify is to flip this one around. The further up or down now in the pyramid, you are the more decision-making power you have and the more you should try to delegate to the people closest to the actual work. By doing so, we can distribute the capacity, and hopefully utilize the full company. Super easy to say, very hard to do.

The reality is that management has way more weight in the organization and we need to think about ways in which ow we can level engineers with managers. We've done that in a lot of ways in Spotify, both when it comes to our career frameworks and leveling the manager track with the IC track, and giving engineers a seat at the table at all forums where decisions are made and so on. I think that's a pretty big part of it, but it's also through how we actually distribute decision making in practice.

I found this model that I really like, which is from Shane Parrish in the "Farnam Street" blog, which is a great resource if you haven't checked it out, I really like it. It's talking about decisions on these two axes of consequential and inconsequential. What is the consequence of this decision? How severe of a consequence is it? On the horizontal there, how easy is it to reverse this particular decision? Depending on how you categorize it, you will act differently. What he is talking about in this blog post is, for himself, as a manager or as a leader of this company, at the bottom he definitely tries to delegate everything. If it's a more reversible type of decision, he would expect the team to gather more evidence, and he himself likes to spend more time deciding when it's both high consequence and hard to reverse.

If I translate this to Spotify context, just to give a few examples that I've been thinking about creating there, let's say we have a few decision points here, and we can plot them on here and think about them as, what is the actual consequence of this and how reversible is this particular decision? Based on the answer to that, we can think about then how should we distribute this. Should I just make it in isolation? These examples would be, let's say, I've come to the stand-up in the morning, should I be pairing on this task today? It's at the end of the day, it's not reversible. I did it or I didn't do it. The consequence of that particular decision is not that high. I missed the learning opportunity potentially, I did something else instead.

Another one. This one we can argue about the consequence, I suppose, but can I merge this pull request? Normally, if we're doing our job well and we have a lot of automated tests and it's a small commit and so on, it's shouldn't be that high of a consequence. Now, we might have different experiences of that, so need to decide where to put it. That's normally also something that we should be able to make pretty quick and we probably don't have to distribute it to a huge group of people. A hiring decision is a fairly high consequence. I would argue it's reversible, but it's not something we really want to do. It kind of sucks, I’ll put it there.

The last example here. Shall I introduce a new data processing framework today? It's a pretty high consequence depending on the size of the organization. I suppose at Spotify, it would be quite expensive if you add another one, and sure it's reversible, but it would be quite high cost. If you're looking at then these different tools, like where do I create the DIBB, or where do I write up an RFC around this, I would probably not do it. I wouldn't write a DIBB if I want to pair on a task today and share it with 200 people. However, if it's about introducing a new data processing framework, it probably makes sense to circulate that with a larger group of data engineers and people around the company and see if people have concerns around it. That's a neat little tool to think about. I like these two by twos, it's always easier to remember than more complex decision-making frameworks and so on.

High Consequence Decision

To give you a few examples of how we approach, I think the more interesting decision points here are the high consequence decisions. I have a few examples of those and how I have approached them. The first one here is on crafting strategy, which is a really interesting one because in my experience, this is often something that management or product management will do, and engineers don't always get involved. A lot of engineers don't want to be involved in this in my experience. I think a lot do, but I think it is really important.

The story here is, when I joined the team in San Francisco we had a bit of an unclear plan going forward. The first thing we started to do was just look at what are the metrics? What is the mission of this team? What is it that we're trying to do? I realized that there was a big lack of alignment across the team and figured that we can do this in a couple of different ways. We can have a couple of managers or product managers sit in a room and craft this and then try to sell it to the team. But I thought, if we can do this more collaboratively and do this in a way that creates the buy-in, we will also drive a lot of motivation in the team.

What we did was we just took the whole team out for a few days and we started from scratch with the metrics and the mission. What is it that we're trying to do? We had an idea. We were in the parking space, but we really crafted a new mission statement and clarified what are the metrics that we're going to measure ourselves on, and that the rest of the company can measure ourselves on? This would then form the contract for us and with the rest of the company. Then we started looking at, if these are the metrics we want to move, what are the big things we can do as a team to start moving those metrics? You see the pyramid there on the window. It's our high-level strategic roadmap where stuff that's further down is a bit more detailed and closer in time. The higher up it is, it's bigger chunks of stuff.

The moral of the story is that after this I saw a huge change in the level of motivation in the team and people started to engage really deeply with this mission. A particular thing I remember is coming back from a weekend on a Monday, and I was talking to an engineer on the team and he said, ''I have this idea on how we can move this particular metric through an integration, in this case with Facebook on the Messenger platform. If you type in "happy birthday" on Facebook Messenger, it would suggest a song, a happy birthday song.'' He thought that that could drive a lot of new registrations through that platform, which sounded like a cool idea. We're, ''Yes, let's talk about, let's look at how we could implement that.''

Then he goes, "I already deployed it. It's already out there, it's launched.'' We had to talk about that a little bit, but he was extremely bought in to this mission and the strategy, and stuff like that started to happen. We needed to pull back a little bit to have a bit more control of some of the things, but that was really cool to see. It was mostly engineers in here, if you do get the opportunity to partake in these sort of strategic discussions, I highly recommend it. If you don't, try to kick in those doors and get a part in those discussions. I think it is extremely useful for the company.

The second example here of a high consequence decision that quite often might happen a bit in isolation, is a reorganization. Who's been in a reorg before? Who's enjoyed it? There are a few. It's always tough, I'm not 100% sure that everybody who was part of this example would have raised their hand. I think a lot would. We had this problem, this was three years ago. I was in Stockholm working at the time and we had a group of roughly 150 people, a little bit more people than in this room, that had been in and working in these teams and tribes that were quite adjacent, but they had all grown quite aggressively over maybe the last year, and the missions were starting to get super polluted. We're looking at it like, "I don't get it, I don't get why we're a tribe. I don't get why this is a team over here. These two teams are doing the same thing. This is getting really messy." The nice little diagram I had in the beginning there was not there at all.

Then I was tasked with solving the problem and reorganizing all of these people. My instinct was that I want to do this in a way that gives people the chance to have a say in what it should look like and in particular where they're going to work. I don't know if you've seen some of the earlier talks today, but there was the one on self-selection this morning, which is sort of what we did here. I'll skip the first part of it, which was coming up with exactly the tribe structure and squad structure. We did that in a very inclusive fashion, but perhaps the most interesting part of this is when we did the actual staffing of it.

We spent a fair bit of time. We have these 150 people in an organization that didn't make a lot of sense, we created a new organization collaboratively, everybody had a say in this. Then we got to the staffing part, which is where - in some places you might sit with a spreadsheet and think about how do you create the perfect team. These people would be the perfect team over here and so on. We just drew this new structure on a big whiteboard. Then we set a bunch of constraints on each of the tribes and teams, like how many people need to be in each team, roughly, at least. This is a team building a mobile feature, so there need to be mobile engineers in this team and so on, and it all added up. Those constraints were there. Then we put everybody's name on a sticky and put it next to the board. Then we gathered everybody and said, ''We have a week when this needs to be staffed. You're solving much harder problems every day, so please solve this one as well.'' This was not necessarily a super easy decision to make to do it this way. I had some people around me that were challenging me quite hard on it. I thought we can do it. Some people wanted to do it a bit less transparent.

We went with this and spent the week on it. We had a lot of stuff happening around it. We had a stand-up by this board every morning. Every manager had essentially daily one-on-ones with people in their team to make sure that people weren't intimidated by this or felt lost. We really tried to catch everyone, and I think we did. We ran into some problems that we had anticipated, like one team that nobody wanted to be in, for example - it was quite an important mission for the company - and some other that was quite overstaffed because it was pretty cool. Then we just got all the engineers who had previously been owning what that team was supposed to do and some other folks into a room, and said, ''Ok, what do we do?" That was a pretty easy one. The response was just, "You created a maintenance team here. Who would want to be in that team?" Let's just slice that, give it to these three teams, problem solved. Easy.

We had a few things like that, but at the end of the day, it went really well. I think people were super involved in this. When we left this thing, nobody had a feeling of just being moved to a team that they had no part in deciding. That was cool.

I have one more example here which is, what technologies should we use at Spotify? This is on a bigger level, what programming languages, what processing frameworks and technology should we use? Here, if you remember TAG, which was the technology wide architecture group, they own a tool that we call the Spotify TechRadar. This is inspired by ThoughtWorks who did something similar. What you see here is, at the center of the circle it says use, then trial, assess and hold. This means that what's in the middle there is programming languages, tools, frameworks and so on, that we have agreed that this is stuff that we use. Trial means that it is in use or we are testing it out, but we're not sure whether it's fully committed to. Assess means that we're looking at it. We're not really trialing it yet. Hold means that we're not using that right now. This could be anything, a programming language can be in there.

How this works in practice is that any engineer in the company can just do a pull request towards this tool and he TAG group will review it and get back to that engineer and there's a discussion, and either it's in there or it's out. If it's out, there's a good reason for it or there has been a discussion about it. The engineer might write up a DIBB or an RFC around why they think this new language or tool should be in there, so that the TAG group has more information to make the decision from.

Isolate and Move Fast on Low Consequence; Distribute High Consequence Decisions

The last principle to help tear down the hierarchy is to isolate and move fast on low consequence, because I have to say that that is quite often the case. I don't know if that's happened for you, but when I work with teams, sometimes we get stuck in over-indexing or solving some problem that's actually quite unimportant. Then we make a really important one way too fast. Move fast on the low consequence and distribute the high consequence decisions.

In summary, optimize for autonomy and enable alignment across the organization. Don't tell people what to do, tell them what you need. Lastly, be intentional about how you make decisions and invest in the ones that matter.

Questions & Answers

Participant 1: For part of your talk, you were talking about getting the team involved in choosing their own team. I'm just wondering how long did that process take, as sometimes we try to be inclusive but what ends up happening is it just really drags on. How do you deal with that?

Lindwall: The question was how long did the reorganization process take? I've done this on very different levels of scale. I've done this on the 20-person scale and this is the biggest one I've done, 150. Twenty people - a few days, a week. This one, the process leading up to the actual staffing probably took six to eight weeks. Had I done it again, I could do it faster, I think. There are a couple of learnings there. I think you can do it in a few weeks if you devote the time, and then the actual staffing part was one week. All in all, I'd say that took two months. I believe I could cut it to a month maybe.

Participant 2: How did you solve the problem when no one wanted to be on a team? You told the story of when engineers decided in which team they wanted to be and then one team, no one wanted to be in. We had the same problem, we couldn't make a good decision without forcing people.

Lindwall: The question is what we did with the team that nobody wanted to go to. As I described, for that particular team we brought everybody together who had knowledge of the problem space and they came up with a suggestion; in this case that particular mission was like we had done a poor job and nobody had really spotted that. Even though engineers had the opportunity to challenge that mission, nobody had before we did the staffing. Then when we did the staffing, we realized that. Then in this case, the engineers in the teams that knew most about the problem said, "This is super uninspiring. Let's just re-slice the pie in a different way here." If that's a possibility, then that's what I would suggest. If you can't do that, then maybe try to sell the mission in other ways.

Participant 3: You had up there the tech radar and the one group makes decisions about what technologies you can use. In this distributed decision making model, what do you do when a team or a tribe decides to not follow that? They decide to pick a technology that is not in there.

Lindwall: What happens when people don't follow the tech radar? It happens all the time, and frankly this is a solution to that problem. If you go back - maybe two years old - if you go back past that, we were really all over the place. This is one of the ways we are trying to get people to align. What happens if somebody doesn’t use it? I think it has to be local. We are quite distributed and autonomous, as I've been talking about, so what I think I need to do - right now I manage a 60-person team here in New York, in that part of the organization I need to build up a culture where we actually leverage these tools and look at them. Where the management - in this case management or more senior engineers - I would leverage as a group to get together, maybe, the TSG and the tribe, to look into what every local team is actually doing. If they're introducing a bunch of stuff that is super misaligned with what we want to do, I would just take the hard conversation then and there. I think it's about having some pretty close precedence of a mix of staff engineers and management to spot the problems, and be able to challenge engineer. I think most people buy this, this is created by the engineers in the company. There's generally quite a big large buy-in into the tool and that I would say so. That would be my approach.

Participant 4: How do you manage initiatives that are multi team in nature, for example upgrading to a new version of Python, which is something that could touch many teams and there are some benefits from or some efficiencies from getting multiple teams aligned?

Lindwall: How do we manage a project or initiatives that span multiple teams? There are two things to that, I would say. One is upgrade to, say, Java 11, we just have to do that and we have to do it across the board. Then there's typically a part of the infrastructure organization that is driving that. It is setting some expectations on the tribes out there. This is the timeline now and this is the reasoning, so just get it done. Then it's up to every tribe to get on that and get it done, and it's usually somebody who we call road managers, somebody in every tribe who takes on to drive the Java upgrade, for example.

The other answer to the question, if it's more of a project as in feature development and not necessarily an upgrade, there we have other mechanisms where the most prominent one, I would say, is what we call company bets. I talked about bets and John [Cutler] earlier talked a lot about bets. That would be a big initiative that we want to get the company behind. It's something that typically spans 10 or more teams and there we have a companywide priority list and mechanisms to get teams behind those initiatives. That's one way of getting alignment around some things in parallel with the more autonomous working model where we're solving for the more local problems.

Participant 5: How do you determine who is on your architecture group? I am in a smallish company and right now we just have everyone on the architecture group who wants to be on it, which can get really messy. As we've been growing, I at least see that we need to limit that number of people, so that decisions are made by people who understand what those decisions actually mean. In a company that's much larger, how has that worked out?

Lindwall: How do we decide who's on the architecture team? As I alluded to, we have a career framework where typically staff engineers, senior staff engineers, and principals are amongst the people who get involved in it; not all of them, but rather we try to get a mix of people from a broad range of tribes. I want somebody from the mobile side, somebody from web, desktop, data, ML and so on, and get the group together.

Diversity is incredibly important for us as well. We do try to mix it up through that lens as well, and then keep it to a somewhat reasonable size. There's a chair right now, one principal engineer in Spotify who's chairing that group and he's driving that process, but that's the selection criteria. In the more local TSGs, same thing. We try to get representation from the squads in the tribe if we can. More senior engineers are likely the people we really want to be there to help facilitate decision making. To be clear on that, it's not necessarily them who at the end of the day make the decisions; they're driving it. Other people might come and talk and work with the TSG or the TAG group to get to drive their thing, but they're there to facilitate it.

Participant 6: Just one question about the reorganization. Did you face any people who were strongly against the reorganization decision? If you faced this, how did you guys solve it?

Lindwall: Did we face any people who were against the reorganization? I don't think we did. Wwe did this, and frankly I think we did this probably too late. If I would have done it again, maybe we should've done it sooner, but maybe because we did it so late, we had the buy-in because it was starting to get very obvious. We explained it. If you look at these two teams, they have nothing to do with one another. It doesn't make sense for them to be in the same tribe, and talk you through these problems. A lot of these problems came from people in the teams to begin with, so the buy-in was pretty much there. We didn't get a lot of push back to do the actual reorg. We did get some feedback after from a lot of people on different areas that I can talk more about.

Since I mentioned that - really quick - the strongest feedback actually was from management, because we ignored them a little bit. We thought too much about the engineers and said we'll deal with managers and product managers later. The whole self-selection managers were actually not included, so they were quite stressed. They were, "So what about me? What am I going to do after this?" We've all figured it all out and it was fine, but it was a bit of a stressful period, so I wouldn't have done it that way if I did it again.

Participant 7: Yes, but that's just my question because [inaudible 00:49:26] manager role and the power to [inaudible 00:49:32] so if you don't involve them how can we process the change?

Lindwall: How can we process the change? There was the before world. We can chat a bit more later and I can talk you through if you want to hear more about it.

Participant 8: In the end though, all those engineers they work with a certain foundation team, which is kind of a shared resource for the entire organization because you don't really have a distributed foundation team. How do you allow engineers to impact what's going in the lower level of the infrastructure of the company or the foundation team?

Lindwall: How do we allow engineers to have an impact on lower level technical decisions around infrastructure? To begin with, we have quite a lot of teams working on those problems; several hundred people are working on the infrastructure and making decisions in that space. One part of the answer is, it's quite easy if you want to move teams in Spotify. It's not that hard, because most teams are always hiring in Spotify, so there's usually an opening. If you just have the right conversation, "I'm really passionate about working with machine learning infrastructure" for example, you talk to that team, talk to your manager, you figure out you can go there and work on the problem more permanently. If you just want to check it out, then maybe address a part of it.

We also do quite a lot of embeds. We have the [inaudible 00:51:28] for teams, as part of my reorganization here. We have quite a lot of people coming in from, say, Stockholm, and work with us for two, three weeks because they're particularly interested in some part of it. They work on that problem for a few weeks and then they go back.

Then lastly, I would say that we have this whole RFC DIBBs process. If one of the themes in my org now is making a bigger change around data infrastructure, say, they typically write up an RFC and share it with the data engineering community. If you're in that, which is open for everyone and have comments and want to influence what they do, you can go in and comment and challenge them or propose stuff to them.

 

See more presentations with transcripts

Recorded at:

Aug 18, 2019

BT