Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Scaling Dilemmas and How to Deal with Them

Scaling Dilemmas and How to Deal with Them

This item in japanese

Mary Poppendieck gave the opening keynote about scaling dilemmas at the Agile Adria 2015 conference. Making teams working together can be challenging, but it is often needed to develop and deliver large complex products. In her talk Poppendieck presented ideas for organizations that want to scale agile.

InfoQ interviewed Mary & Tom Poppendieck about balancing cooperation and autonomy in agile teams, doing experiments to deal with complexity, how a product mindset can give better results than a project mindset and how you can focus on the business impact.

InfoQ: In your presentation you covered four problems that scaling tries to solve: Cooperation, complexity, mindset and focus. You mentioned that teams are looking for ways to balance cooperation and autonomy. Can you elaborate what makes that so difficult?

Mary & Tom Poppendieck: In his 2009 book "Drive", Dan Pink emphasized three things that motivate people: Autonomy, Mastery, and Purpose. People in the agile movement took the word "autonomy" to mean, quite literally, "freedom from external control or influence; independence." Interestingly, the agile movement has generally applied autonomy to the small team level, rather than the individual level, despite the fact that biases like groupthink often reduce individual autonomy in teams.

In many places, it became popular to believe that small, cross-functional development teams should be completely independent – with little attention given to the fact that these teams are part of a larger whole which must succeed if the small teams are to succeed. This focus on small autonomous teams has often turned into a feeling of entitlement, so when teams are asked to cooperate with each other to achieve a larger goal, they frequently feel like their autonomy is being compromised.

The fact is, in order to cooperate, people and teams must make accommodations to the needs of others – they must give up focusing exclusively on their own goals in favor of a larger shared goal. When it is possible for teams to independently be responsible for a piece of code – a microservice, for example, there is less need to make accommodations. But in many domains, it takes a lot more than a handful of people to develop a module or component, and these people must work together as a larger team – or a group of small cooperating teams – where autonomy applies to the larger group.

Large cooperating groups are certainly not a new idea – in fact, teams of 30 to 50 people cooperating together have been common for a very long time. Evolutionary biologists call groups of this size a "hunting group" – the number of people necessary to achieve a larger purpose such as hunting large prey to feed a village. Nobel laureate Eleanor Ostrom has documented many instances of large groups cooperating to effectively use natural resources such as forests, pastures, irrigation waters, and populations of fish.

Unfortunately, agile software development has sometimes gone a bit too far in defending the autonomy of small teams of about 7 people, without practical ways to talk about how these teams can accommodate the needs of other teams to achieve higher level goals. So we need to tip the balance away from autonomy based on small teams and toward cooperation in properly sized groups working toward a common goal.

InfoQ: Can you give some examples of how people can effectively cooperate in larger groups of 30 to 150 people?

Mary & Tom Poppendieck: It usually takes a lot more than seven people to put a new product on the market and make it a success. It doesn’t matter if it is an insurance product or an app for a smartphone, you will find that insight from design, marketing, engineering, distribution, support, finance, and many other areas are necessary to make the product a success. Look behind a few of the great products you have encountered recently and you will probably find a team that is thirty to fifty people strong.

We have the (mistaken) idea that a delivery team is an autonomous team – but it’s not – it’s generally a sub-team of a larger team. By breaking off software teams into small "autonomous" teams that are told what to develop, we have limited their contribution to the whole product. I believe that great products evolve when the engineering team is part of the larger discussion of what features a product should have, what technology it should leverage, how it might be presented to the market, what support is needed, what evolutionary path would work best.

Teams of 150 are common at Gore and Associates, where they comprise an entire business unit whose members’ livelihood depends on the success of their business. Teams that developed one blockbuster animated film after another at Pixar had about 200 members who worked together for three years, focusing on how to help their teammates (often in other disciplines) turn out their best work.

InfoQ: You suggested to do "poke" to deal with complexity in scaling. Can you describe how and why this works?

Mary & Tom Poppendieck: Complex Adaptive Systems Theory is useful for thinking about scaling software – especially software that is intertwined with products, markets and businesses. The entire software-intensive business system is clearly a complex system. One of the things we learn from complex adaptive systems theory is that leaders of successful systems constantly adjust the balance between agency (self-organization) and connectedness (interdependence of agents) depending on the context. This speaks to your first question about the balance between autonomy and cooperation.

Complex systems that grow and survive over time are adaptive, and that adaptation occurs in a particular manner. A steady stream of small experiments – many of which do not work out – cause the people responsible for the system to gradually discover better ways of doing things. Small experiments are necessary because even if the complex system is deterministic, we know that tiny variations in input conditions (e.g. the flap of a butterfly wing) can cause large variations in the system response.

Big changes to complex systems are guaranteed to have an effect – but predicting what it will be is virtually impossible, because no person or team can understand the intertwined dependencies. Therefore anyone who values predictability and safety should understand that the only way to achieve stability it to try small things, observe the consequences, and adapt the system to deal with them. The idea that today’s systems are predictable and that big, well-planned changes are a good idea has long since proven to be wishful thinking. We need to understand that just as in war, in complex systems, planning is useful, but plans are useless.

InfoQ: Can you elaborate how a product mindset can give better results than a project mindset?

Mary & Tom Poppendieck: The problem with projects is first of all they are big batches of work and secondly they are aimed at following a plan. In a lean software development world, we have learned that deploying small batches of code leads to dramatically higher quality as well as a huge increase in flow efficiency, (and thus overall efficiency).

In projects a so-called "delivery team" is expected to execute a well-crafted plan and is measured based on variance from that plan. But this assumes that the plan – which was made when the least information was available – remains valid throughout an extended execution period. It also assumes that learning more about the situation and adapting to it is not necessary. Some larger projects tolerate rolling change for future parts of the project, but fundamentally projects are built on the idea that plans are valuable and should be followed. This is a flawed assumption in an uncertain environment.

Products live in the world of uncertainty – economic uncertainty, competitive uncertainty, impact uncertainty. They are conceived of, experimented with, and fielded by a diverse team that includes design, product management, marketing, engineering, support, and operations. When these various competency areas work together to understand the market, the consumers, the technical situation, and the future possibilities, great products emerge. In fact, there are almost no great products on the market today that did NOT emerge through the process of experimentation.

Marty Cagan (from Silicon Valley Product Group) claims that over half of all good product ideas for software-intensive products come from the engineering team, because these are the people who understand what the technology is capable of and where it is heading in the future. This means that when engineering teams that are focused on project delivery based on what "the business" asked for, the majority of potentially good ideas will never make it into the product.

InfoQ: You talked about focusing on the business impact when scaling agile. Can you give examples how this can be done?

Mary & Tom Poppendieck: I would like to rephrase your question – why would anyone want to "scale agile"? When done correctly, lean (and hopefully agile) is a mindset; it is not something you measure or scale. What you can scale – and measure – are the positive business impacts brought about through the adoption of lean and agile principles. So let’s talk about why and how to focus on impact.

Smart, creative engineers will find purpose in their work. If the structure they are confined to does not enable them to really make a positive improvement for the people who benefit from their efforts, they will find purpose in the opportunity to explore neat technology, to enhance their resume, or otherwise pursue an individual or local team vision. But engineers are people too and such inward focused purposes are not nearly as motivating as is applying their technical mastery to effectively solve a problem for other people. Solving a significant problem for people you care about in a way that contributes to a sustainable business impact is the kind of purpose that best generates self-motivation.

How is this done? The most important thing to pay attention to is the length of the feedback loop for a designer, product manager, or engineer – or preferably all three working together. How long does it take from the time they decide to try something until the time they receive feedback on whether the idea – as implemented in practice with real customers – achieved the kind of impact they expected? Is this feedback loop hours? days? weeks? months? never? Designer Bent Victor has a guiding principle: Creators need an immediate connection with what they create. We have a similar guiding principle: Teams should adjust what they are doing based on what team members learn directly from their efforts. Shorten the feedback loop.

Rate this Article