This is Deborah Hartman and I am at Agile 2008 with Bas Vodde who is co-author with Craig Larman of the new book "Scaling Lean and Agile Development. Successful large, multi-site and offshore products with large-scale Scrum". Welcome Bas. Could you tell us please about what it is that you do over in Singapore?

Well currently I am writing a book. And I coach and train companies on Agile adoption mainly, mainly larger companies and mainly larger products, larger legacy products, and then also the company transitions. And I try to find most of my work, or I work mostly in Asia right now.

So are you excited about the book that you have written? Tell me why.

Absolutely. There is not very much written on large scale Scrum, or large scale development and most of the things that have been written are limited to only a couple of projects or products or companies, because one of the challenges with large scale development is that large projects or large product development is slow and they adopt things slower and large companies tend to be slower so the effort is hard to get experience from multiple products and especially from multiple companies.

But in your case you have some of that: you have multiple companies as examples in your book?

We have multiple companies and multiple products in multiple companies. So I worked closely with a few products and then saw a couple of other products in a couple of companies and then Craig also worked closely with one or two very large products and then also consulted on a couple of other large products but not that involved, because it takes a lot of time to be involved in large product development.

In the face of the fact that adoption is slower in these large product development efforts, how is your book going to assist organizations that want to make this transition?

Well it's going to tell some of the experiences we had and some of the organizational structures and setup which we think is needed when you scale up large development.

Your book title mentions something called "Large scale Scrum". Is this a new methodology?

No, it's not a new methodology, when you look at Scrum, Scrum by itself as it is scales up to whatever size products you have, you don't really have to change very much about Scrum, but you have to figure out how to do certain things in parallel and how to work with a backlog with multiple teams and how to organize the multiple teams and that kind of things. So large scale Scrum is basically normal Scrum with a few of these additional guidelines, of what to do, like have featured teams and have one product backlog and then we also have some recommendations for how to work across if you scale larger than a hundred or two hundred people because then it becomes harder to manage the product with only one product backlog.

I have seen organizations turn their teams Agile and create Agile component teams but have a heck of a time trying to coordinate them.

One of the key messages in the book will be that, and is on the InfoQ website right now as an article is that, and this is one of the harder changes, is to try to change from component teams to feature teams where component teams the standard teams where the teams are organized around your architecture of your system so a team owns a component of the system. And they will need to move to feature teams where a team gets a complete feature and makes the changes wherever they need to do that in the system.

If you move to feature teams then a whole bunch of the coordination between teams falls away or at least coordination between teams based on the requirements goes away and the coordination between teams is more based on the code, and if it is more based on the code then Agile practices like continuous integration help a lot and kind of resolve some of the communication problems.

That is interesting. So tell me about this communication across teams that is more technically based.

If you got a bunch of component teams and you got one requirement and one requirement will never fit to one component team, and therefore the requirement will need to be split up and parts of the requirements will need to be given to different teams. Once you got a requirement and you give it to multiple teams and the requirement is over, first of all there will be a lot of communication freezing of interfaces and that kind of things, to decide, because interface in software and communication between teams in component teams is equal to each other so teams need to freeze their interfaces because their interface is the communication between other teams.

If they freeze the interface they can program to the interface and then hopefully later integrate it together. And because the requirements never balance to the teams, therefore the teams will get out of synch so when they got maybe perhaps not the highest priority requirement then that requirement will be worked on for example in one component team in one sprint but the other team will not be working on that requirement yet and they will be working on the next sprint and therefore you get not only the communication or the synchronization issues but because they are not working on the same feature at the same time you get this other team sprint later communicating back about one thing that they did in the previous sprint. And you get kind of the same problems as traditionally between developers and testers, where developers are annoyed by testers because testers are giving feedback about what the developers did earlier.

They are on to something else new.

And so they don't want to be interrupted and that is what you automatically get to a component team so they are not between developers and testers but between one team and another team. And of course you got the question of "Who needs to split things up in the beginning and who needs to integrate it in the end or test it and integrate it in the end".

So what does technical communication across teams look like with feature teams rather than component teams?

When you move to feature teams you give a whole feature to a team and therefore you don't have this timing issue. The only thing that happens is that multiple teams will be working on the same source code, on the same components. And therefore they need to manage their changes in that component at the same time. So the coordination will need to be about "I do this in this component" and the teams will need to talk about especially creating the components in a good structure and removing duplication and that kind of thing. So discussion between the teams is technical about how the code structure and how the component structure would be instead of related to the requirements and the interfaces. So you don't need to freeze your interfaces anymore except when you publish your APIs because most of the interface changes will be within the same team.

How do these conversations across teams fit with the rhythms of Agile? Because each team has features and they have tasks that they have planned out during the sprint but it sounds that also during the sprint they also need to have these conversations across teams about how they are working in the code base?

One of the key practice for letting multiple teams work on the same component is continuous integration so a lot of the communication actually goes by the teams via integrating lots of small parts. So for example if one team makes a change in a component the other team is also working in that component and they notice that the other team makes that change then at that moment these teams would talk to each other and that is another important concept which we talk about in the book is that teams within a large product need to be responsible for their coordination between teams. Instead of creating a role for coordination, it's the job of the team to coordinate between teams and such coordination could be changes in the code for example.

So, inter-team communication becomes part of the self-organizing pattern of Scrum.

Yes. And one of the reasons for doing that is that if you go back in the team research of the last many years then one of the critical success factors of well working teams is that the teams manage their own boundaries. So it is called sometimes boundary management where the team itself takes the responsibility for their context within the organization or their context within the product. Traditionally a lot of team-thinking is trying to create a sandbox for the team so that they can focus on a particular thing. But according to team research it shows that that doesn't create well working teams but what creates well working teams is teams that are internally focused and at the same time externally focused and creating or managing their boundaries between the teams themselves.

I have worked with groups that their first reaction is that it is impossible to move from component to feature teams. They say we have too many specialists we need to share people across teams, what's your experience with making the transition?

Exactly that. The transition is really hard for most organizations because it involves a lot of learning where people are very specialized and in most large products people are very specialized on one component and in traditional development people didn't recognize or they did recognize that it was a problem, but it wasn't very visible, but once you start creating one product backlog for a product and start working with component teams then you start noticing that the coordination and the planning and that kind of things are almost impossible to do.

My experience is that it's the same, most organizations find it almost impossible because as you said they are very specialized around their components and not only does it mean to basically reorganize and reform teams which is hard for most organizations it also means that all of the people would have to learn a broader area and a lot of developers are comfortable with having their part of the code so then once they have to start working broader or once other people start working in their code they become very uncomfortable and also, and this is quite interesting, their productivity drops so they feel that they are less efficient and they are probably right that they are less efficient in for example writing a certain amount of code.

But if you have component teams then part of the definition you will locally optimize your efficiency only to be efficient within that component team, where you want to optimize end to end, and therefore when you move to feature teams, it tends to optimize end to end but locally people feel slower. The interesting things, like when I work with products then I normally try to look at their code because if you look at code in large products it tells so much about what is going on in the product. All the problems in the organization and all the miscommunications and lack of scale and whatever is there, is visible within code.

And what happens in traditional component teams is because everybody has their own sandbox, therefore the mess people are creating in their part is not visible. So when you move to feature teams, which is basically a shared code ownership on a very large scale, then that problem, the bad code of other people and the disagreements about coding style and those kind of things, they become immediately visible. And that is also one of the large problems which people feel uncomfortable with. I was just talking within an ex-colleague of mine yesterday and he mentioned that when his product moved from component teams to feature teams for the first time they saw the amount of crap that was created in their product before but not that he had ever noticed this because it was hidden in "my code" type of thinking.

You now have teams sharing code that previously was kind of private inside of the component team. And you have people starting to learn skills that they didn't need before. And to have to train other people in skills that were specialized before and it sounds like the team is substantially going to slow down.

Yes, when an organization moves from component teams to feature teams they tend to slow down in the beginning though interestingly if you look at it from the product backlog prospective so if you would take all the requirements, put them in priority order, then traditionally because of the organizational structure people would be working on lower value functionality. Even though the teams slow down, they are more focused on the more important things and therefore sometimes we say it is more important to develop low value features fast or high value features a little bit slower. And probably the last one is a better choice in most cases.

What happens to moral? People must be nervous when this shift is proposed, when it starts. What happens to moral over time as this transition comes in?

It depends very much per person. Some people are uncomfortable giving up their sandbox, they were comfortable being given some piece of work and they could just figure it out and do it by themselves, and they feel uncertain when they have to edit others people's code or things they don't know very well and then there is of course, the problem that most of the large products test automation, which makes it very painful to edit other people's code because you don't know what it does exactly so it's for you to write tests first and people are not used to writing automated tests in most large scale projects.

And al of these things they get together in this transition. So these people they feel uncomfortable and uncertain and interestingly enough some people feel that they lose their skill. I always see it as the opposite that they gain much more skill, but they consider that they are the experts in that area, so they put some self worth on being an expert in that area. And when they have to learn more they feel the loose skill even though they are learning more.

Some people find it really challenging that they transition. And others?

And others they find it really exciting because and in my experience a lot of the good developers which are stuck in large products they like to learn and they like to see more areas and because they were put in this organization, organized around code they were kind of stuck, put in a box and they couldn't really get out so a lot of people they are quite excited about learning much more in those areas and trying to really understand what the whole product is doing.

Because one of the problems in very large products is that unfortunately many of the developers they don't really know what the whole system is doing and when they start learning and look at other components around it and especially when they take, because it is very often impossible for everybody to know the whole product, but when they take a feature focus, at least they know how this whole feature works end to end in that product and some of the people will switch from component to feature teams, they feel quite excited about that. Finally they can get this whole feature to work. And that goes back to teams and teams motivations again we have also in the book a chapter on teams and what motivates teams.

And one of the things is the end to end responsibility and the feeling that you are doing meaningful work. When you are just a developer having one particular component it's hard to see how you are doing meaningful work or how your work fits within the whole picture. Once you move to feature teams, you can better see that, although it would put an enormous burden on learning, so people who like learning are comfortable and excited with it.

Tell me what shape that learning takes in organizations as this transition from component teams to feature teams rolls out. I am thinking there are classes there are books, there are study groups, informal learning. What do you see working well?

In traditional organizations you still have components basically so obviously a lot of the learning will have to go from these specialists teaching other people and that can be by pair programming or by design workshops, things like that. And the learning is actually not as large as people suspect it is, because this is one of the criticism people sometimes have, that in feature teams it seems like everybody is a generalist, everybody can do everything but in fact that is not true, when you got for example a team with people who are specialists in a certain component, then most likely they will pick up a feature which was seen in that component or in this component so therefore they are still using their specialists skills when they are implementing this whole feature.

And the same also inside a team is not that the whole teams needs to know everything but as a team together they will need to know at least all the components that this feature will touch. So therefore if I am a specialist in say component A and there is a piece of the feature that makes changes in component A then in most of the teams I would still do that of course. So the real learning occurs when there is a mismatch between the skills of the team and the feature they are given, or when the skills in the team do not balance equally with the skills needed for a certain feature. For example there is much work in component A and little work in component B so therefore the person who knows component B will need to learn a little bit about component A. So therefore some of the learning is actually quite gradual and this is interesting that the greater the mismatch between the higher priority of the features and the skills of the people the greater the learning will be and that is exactly what you want.

So you have talked about teams working in the feature team configuration, whereby communication is largely through the code base and through the design of what they are working on. How big can you scale that?

Good question, so not only is the shared code a problem because when you scale up Scrum you want to keep one product owner, one product backlog because you want to keep the visibility on the priorities, so another question you should have asked is "With how many teams can one product owner and one product backlog still work?" because normally you want to have say four backlog items per team per iteration and you want teams to finish complete backlog items in iteration. So if you had let's say ten teams, four backlog items per team per every sprint, then the amount of backlog items managed for one product owner also becomes very tough.

Typically we see that feature teams working on one product backlog kind of breaks around ten teams or around one hundred people. Then it becomes too hard for people to work on the shared code base, it becomes too difficult for one product owner to still manage one product and the whole structure breaks down.

So, you have got a pattern that you say holds up to ten teams let's say hundred people. But not every product can be done on that scale. How do you get bigger than that?

If you work on larger products then you will have to scale up the feature team concept. And we do that with a concept which we call requirement areas. So if you take all the product backlog items of a say four or five hundred people product, and you would create an affinity diagram, you would group them in different categories, based on the customers so nothing related to the components or the architecture or whatever then in your top level product backlog you would have one backlog with all the requirement categories or requirement areas and based on these you could split into different what we call area backlogs, and that means that one backlog item in the product backlog always fits to one backlog item in an area backlog.

The mapping between a product backlog and an area backlog is really simple and then underneath each area backlog you would have what we call an area product owner which is doing the prioritization and clarification for that particular area, and underneath each area you would have typically from four to up to ten feature teams, you don't want less than four feature teams and as we just talked about earlier more than ten feature teams becomes hard to manage the backlog and also the shared code base. That is interesting because if you scale up this way, then what I just said was that the teams would not have to work on the whole system anymore, and therefore they can still manage to learn. What I just said is that if the requirements mapped to one area backlog so therefore potentially one team still needs to change things everywhere around in the whole system.

But interestingly enough one of the common patters of the large product is that requirements in a certain area most of the times are in the same components or in the same subsystems in the code. So instead of creating specialization around a particular subsystem the feature teams in a certain area still say eighty percent of the time change the same components and only twenty percent of the time need to go out to the rest of the system so therefore they can still specialize and therefore it still scales up. It scales up feature teams to whatever size basically.

How does a product owner communicate with ten teams in a sprint?

You need to ask what is the important role of a product owner so if you got one product owner then if you look at the things a product owner does, then prioritizing and value estimating backlog items are critical and one product owner for ten teams can still prioritize the items that is not really the problem. The only thing which is the most difficult to scale is the clarification of your requirements. If you consider the product owner to be the person who needs to clarify all these requirements then it becomes really hard for one person to do that for ten teams.

But the clarification of the requirements actually can be fairly easy done by other people also so if one product owner works with ten teams, then there will be one product owner who maintains the backlog, one product owner who does the prioritization, one product owner who makes the decisions in sprint planning and that kind of things, but not necessarily one person, one product owner who does the clarification of the requirements, so you might have an analyst in the team or an analyst together with the product owner in the product owner team, or a system engineer whomever will help the product owner with clarification of the requirements and the splitting of the backlog items. So that way if you consider the product owner to clarify all the requirements then probably one or two teams would be the maximum, but when you consider that the clarification of the requirements can also be done by other people and that the main thing of a product owner is the prioritization then you can probably do it up to ten teams.

I am interested in this idea of breaking a product into separate requirement areas' backlogs. Can you give us an example of how that would look?

Yes, it depends on the product and typically one way we sometimes do it, if you write very high level requirements on cards and you give it to a product owner and you say "group it" then he will group it in certain areas. So, you take an internet portal might be like sports, news and that kind of things if you take a telecom network element, it might be like network management, performance, or whatever. If you take an huge printer, it might be like pdf handling, etc. so they are all customer focused grouping of requirements and what they are depends on the product and there are will also change over time.

And are these requirement areas prioritized among themselves? What does a product owner do with regard to prioritization across these multiple areas?

So the area backlogs are like a view on the product backlog so every one item in the product backlog maps to one item in the area backlog. So the areas are not prioritized itself but the backlog items inside the areas are prioritized in the overall product backlog. If you take a product backlog, product backlog as product backlog items, you add a category for these product backlogs items and you take out only the backlog items for a certain category and put them in a different list, then you don't prioritize between these areas because you prioritize between the backlog items in the areas.

The prioritization is done in the overall product backlog. So essentially you have teams pulling specific categories of stories of the backlog depending on their areas of requirements specialization.

Yes but what typically happens is that the backlog items in the overall the one product backlog because we always want to have a product backlog, it doesn't matter how big the product, they are typically of a higher level. So the higher level items get into the area backlog they get split over there to smaller backlog items and only when the smaller backlog items differ a lot in priority would the smaller backlog items be mapped back to the overall the big product backlog.

And therefore the only problems that you will have is once teams work in one area, and in the overall backlog the whole product backlog the difference between the items inside the requirement areas becomes too large. And that is the job of the product owner is to decide to move teams between areas. And if a team moves between an area, it means that they will have to learn a new domain a customer domain, so therefore they will go slower, but they will be working on higher value items. And that is the main decision the overall product owner needs to make to decide now we are working on lower value items, do we want to move a team so that their velocity drops they will go slower but they will work on higher items and they will become more flexible.

You talk about as though it was easy to take a product and break it into requirement areas, but I have a feeling that that is not how organizations are structured right now.

No, most organizations are structured on grouping component teams together in what we sometimes call subsystems departments so if you take an architect of a very large system it tends to combine four, five, six, very large systems and that would be on the department level and inside that you would group your component teams.

So is this going to change org structure?

Well just like feature teams normally it's going to change the organizational structure. One of the biggest challenges moving to feature teams, and to requirements areas are not really about learning but about making the decision, to change the organization because that is very hard for a lot of organizations.

Who needs to be on board to make those changes?

Probably the senior management team of the product. Otherwise it's almost impossible to make these kinds of changes.

If an organization is going to move from component teams to feature teams and requirement areas do you have some recommendations for them?

Yes and this is really hard. Normally we recommend to do everything gradual because large changes are really painful but especially in the change to requirement areas, we have seen a couple of gradual moves, but they were more painful then probably moving at once. And we normally recommend to just decide from one day to the other day to go to requirements areas, change the organization and it will be very painful in the beginning but if you do the gradual change it will be probably more painful.