BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Are Your “Value Streams” Keeping You Stuck in the Past?

Are Your “Value Streams” Keeping You Stuck in the Past?

Bookmarks

Key Takeaways

  • The essence of business agility is being able to respond quickly and systematically to feedback. As a means of achieving business agility, value stream management falls short, and ends up being not very different from what organizations have done for a long time: using program management practices to coordinate work across different teams in a large organization.
  • Value Streams don’t encourage empiricism and experimentation; they are simply a mechanism for allocating and coordinating work.
  • The root problem that value streams usually fail to address is that the organization’s “products” are large and unfocused, which requires a large and unfocused organization to develop and deliver. The architecture for these “products” is often large and unfocused as well, and is implemented in disparate systems maintained by heterogeneous teams.
  • To scale their agility, organizations must simplify their products, grow and empower cross-functional teams, and remove dependencies between teams. This requires the organization to master a variety of different supporting practices that help teams to work independently toward shared goals. These practices range from employing a common inner-sourced platform to automating mundane and error-prone tasks to let teams focus on delivering valuable customer outcomes.

You can easily tell when an organizational “improvement” idea isn’t going to achieve much when it can be rapidly adopted by just relabeling what the organization is already doing. It’s not real change, it’s simply rebranding.  

On the surface, the concept of value stream management sounds great: who would not want to produce a stream of value?  But as it’s typically realized, value stream management ends up being not very different from what organizations have done for a long time: using program management practices to coordinate work across different teams in a large organization. 

Value Streams are often a thin veneer over a traditional organization

Traditional organizations are composed of “silos”, or departments that focus on a single set of related skills: application development, data center operations, quality assurance, marketing, business management, risk management, etc. 

In these organizations, “the business” gets an idea for a new product, or a change to an existing product, and like workers on an assembly line, people from various departments work on the idea at various stages in its development until something that a customer can use comes off at the end of this “assembly line”.  

The problem with this “assembly line” model is that it’s slowed by hand-offs between teams, and these teams make decisions based on very little feedback about the value of what they produce (see Figure 1).

They assume that “the business” knows what customers want and so they focus on minimizing the time and effort that they spend producing “value”. But because the organization lacks feedback, it ignores  its greatest source of waste: delivering to customers things that don’t meet the customer’s needs.

As a result, even an optimized “value stream” is often simply producing waste at a faster rate.

Figure 1: “Assembly Line” model vs. empowered, autonomous agile teams

Despite its name, the value stream does not focus on value, but on managing work using a “manufacturing line” metaphor. Once an “idea” is delivered”, the value stream focuses on the next idea. There is little “learning” since ideas flow top-down from “experts” to teams, who implement and deliver these ideas, often with little or no measurement of value delivered and little or no feedback. When implemented this way, value streams are not agile because they never gather feedback, they never inspect that feedback, and they never adapt their approach based on feedback. 

This deficiency seems easy to remedy: simply add feedback. At every release, measure whether the customer’s experiences improved, stayed the same, or got worse, then feed this back into the process by which new improvements are conceived. In this way, the value stream really becomes a loop in which experiments about value are tested. Adding feedback loops helps, but only if those loops are very quick; when they are relatively slow ones that take many months, or more, to get feedback on the initial idea, they are not very effective at improving the value they deliver. 

When the value stream is simply a veneer over an existing siloed organization, the time it takes to go from having an idea to the point where it can be tested is too long to be really useful to run test ideas quickly. If running these value experiments takes days, or a week or two, the organization can make relatively quick decisions about what is valuable, but if they take many months or longer, the organization wastes a lot of time and effort building things that customers neither want nor need.

How this manufacturing mindset undermines agility

The tension between agility and the lean manufacturing mindset behind the value stream concept expresses itself in a number of different ways. The goal of manufacturing is to build identical copies of a standard product, built to a standard specification, without waste or variation. it regards any perturbations from predictability as undesirable defects that need to be erased. Lean manufacturing is an approach that seeks to minimize waste in this manufacturing process.

The goal of an agile approach is to identify problems that need to be solved and developing solutions to those problems. In the course of solving those problems, eliminating waste and impediments is important because it can help a team to focus all their efforts on solving the right problem with a well-constructed solution, but some things that a manufacturing mindset would call waste, like solutions to the wrong problem simply cannot be predicted until the team builds and delivers something to customers and measures the result. Viewed from another perspective, optimizing the way the product is built does not prevent products that don’t do anything useful for customers.

The manufacturing mindset fits perfectly with the traditional organization’s belief that “experts” are best at designing products, based on market research, and informed by their own expertise. The goal of the manufacturing process is to perfectly realize the product definition as it has been handed to them. They may make adjustments to the specification in order to make the product easier to manufacture, but the manufacturing process is not concerned with whether the right product is being built. Again, this fits perfectly with the traditional organization’s belief in top-down control.

The products that an agile approach is trying to solve are generally one-offs. The goal is not to build copies of a standard product, because what the customer actually needs is not well understood. The goal is to discover what the right product is. For some products, the organization may need to manufacture more of these products once they understand what customers need, at which point agile practices are less useful and manufacturing processes are more applicable.

When customer needs are emerging or changing rapidly, a top-down process cannot respond to feedback quickly enough. The kind of process that is better suited to this is a variation of the scientific method - form hypotheses, test them using experiments, inspect the results of the experiment, and adapt the approach based on the results.

Value Streams disempower teams

In addition, the value stream approach can result in disempowered teams, which can lead to apathy amongst team members. Research shows that teams do best when they are given a goal to achieve, the freedom to decide what to deliver to customers to achieve that goal, and the freedom to decide how the team will work to deliver the solution to customers. This creates high levels of autonomy and purpose among the members of the team, which leads to high levels of motivation.

When many teams have to work together in a coordinated fashion, as they do in a value stream, it’s simply not possible to give them, collectively, a broad goal and let them work out what to deliver and how. The people managing the value stream define the solution and allocate work out to teams, although they may leave some of the “how” of working up to the teams provided that giving teams this freedom does not create conflicts between teams.  In very real terms, the teams in a value stream are simply performing work assignments, which results in lower levels of intrinsic motivation than they would have if they had the freedom to define the solution. They may still show high levels of professionalism to do good work, but they lack a sense of purpose that comes from owning the solution.

Why do organizations trap themselves like this?

Many of them know better, but they feel they can’t do any better. Their “products” are huge and cannot be developed and delivered by a small cross-functional team, or even a small number of cross-functional teams working together as in a Nexus. Even when products could be developed by a small cross-functional team, organizational silos and technical specialization create barriers to forming cross-functional teams because no small group of people could be brought together with the right skills to develop the product. 

When this happens, the organizations try to create larger teams with team members shared between teams. Coordinating teams composed of part-time members usually leads them to the value stream concept as a way to coordinate having people working across many initiatives. Organizations are stuck because their products are too big and their organizations are too fragmented for them to work in any other way.  Calling it a value stream doesn’t change the fundamental fact that they have to simplify their products and their organization to get better results.

Value Streams, Products, and Software Architecture

The root problem that value streams usually fail to address is that the organization’s “products” are large and unfocused, which requires a large and unfocused organization to develop and deliver. These “products” are really large collections of different capabilities that might be interesting to a large and unfocused group of potential customers. Their scope is often an entire line of business, which makes it very broad indeed.

The architecture for these “products” is often large and unfocused as well, often cobbled together from many different systems that were developed over many years, sometimes decades, to serve different purposes. These systems continue to be developed and maintained by different teams, leading to a siloed organizational structure. As a result the architecture of the cobbled-together “product” is, by definition, siloed as well.

To tie all these disparate systems and teams together, the organization often creates program architecture teams, which are responsible for the overall software architecture across the value stream teams. Program architecture teams are expected to reconcile and coordinate the various software architectures created by the value stream teams, to eliminate redundancies and to promote common architectural elements. 

Their efforts to tie all these disparate architectural components together often fall short when program architecture teams decide to use a wasteful “up front” architecture approach, driven by quality attribute requirements (QARs) which are based on guesses rather than facts. As a result, they delay the work of the value stream teams, deliver an architecture that may not meet the requirements of these teams, and is likely to have to be repeatedly refactored. 

Alternatively, the program architecture team may decide to wait until the QARs for each value stream are better known to create a program architecture, which results in delays impacting every value stream, and may still need to be refactored as more QARs become known. Either way, a program architecture team approach amplifies cross-team dependencies which kill autonomy, introduce delays and waste effort.  

To get better results, take a different approach

The fundamental problems with the value stream management approach, that it takes too long to get valuable feedback and that, as a result, most product decisions are based on hunches that may largely be wrong, can’t be solved by more tightly managing the value streams. While the speed of the feedback loop can be improved incrementally by focusing on removing waste in the process, incremental improvement isn’t sufficient to achieve the dramatic improvements that an organization usually needs.

Executives often look for dramatic short-term fixes that will solve problems, and management consulting firms are more than happy to sell them these quick fixes. Unfortunately, the desired results are almost never realized.   The solution to the problem that value streams can’t address is dramatic, even revolutionary, but over time horizons measured in years. The organization did not create the mess that needs to be fixed overnight, and the remedies can’t be realized overnight, either.

The journey organizations need to take to become more agile, more responsive, in their businesses involves a lot of moving parts put in place over a long period of time. Briefly, these are:

  • They need to simplify their products. As noted above, many organizations have “products” that are relatively unfocused, doing a lot of things for a lot of different people, but usually not satisfying the needs of any particular group. They often consist of sets of applications that in some way support a line of business, but they are fragmented and incomplete. Trying to make these big balls of mud more responsive to changing customer needs is a losing battle.

Instead, organizations need to create new products that are aligned around the specific unmet needs of a specific set of customers (current or potential). An organization doesn’t need to do this all at once, breaking down its existing “products” into smaller and more focused products; this would be very disruptive and would create a cloud of chaos that would ensure that the entire effort would fail. By focusing on specific opportunities, the organization can get those started on a better path without trying to clean-up the big mud-ball all at once.

  • Around these new products, they need to create and grow empowered, cross-functional, self-managing teams. By doing this, they reduce hand-offs between teams and reduce decision latency, two major factors that prevent the value stream approach from producing the results the organization desires. 

For skills that these empowered teams need but do not have, support them with specialists who can either temporarily join the team to transfer knowledge, or coach team members in the new skills. When doing this, make sure the specialists are available when teams need them, instead of the usual approach of making teams wait for scarce and expensive specialists to find time to help. This may mean hiring more specialists to support teams in the short run, at least until teams can support themselves.

  • They need to gradually break down monolithic legacy applications into smaller products and supporting platforms. This involves a number of techniques and approaches that help teams to slowly create reusable services out of the “big balls of mud” to break-down legacy applications, and to gradually create platforms of reusable code for patterns and frameworks from these efforts and from new development.  As these platforms evolve, organizations need to reward the contribution that teams make to the shared platform to counter the natural tendency for teams to focus only on their own immediate needs. This means creating a culture of technical excellence, shared success, and professionalism that transcends teams and permeates the organization.
  • They need to measure the results teams deliver to customers, and to help them improve their ability to deliver value. This means that teams must deliver to customers frequently, measure results, and improve by inspecting results and adapting accordingly.
  • They need to automate repetitive, mundane, and error-prone manual tasks to let teams focus on delivering value. Any team that is delivering potentially valuable product increments to customers on a frequent basis will be building, testing, and deploying software literally all the time. If teams have to expend manual effort on these tasks they will not be able to gather feedback quickly enough to make timely decisions. Furthermore, people who perform tedious tasks tend to make errors that create waste and delay the work while the errors are corrected. Automating these repetitive tasks helps people work faster with fewer errors. Collectively, these kinds of tasks are broadly discussed as DevOps practices, and teams can’t run effective feedback loops without them.

When an organization consistently approaches new opportunities in this way, and focuses on improving the speed that it is able to gather and respond to customer feedback, it builds its agility product-by-product and team-by-team. There isn’t a short-cut quick fix approach that can replace this steady, consistent approach, but this steady, consistent approach makes continual incremental progress that results in significant improvements over time.

Conclusion

One definition of insanity is doing the same thing and expecting different results, and yet organizations continually avoid real change because they feel it will take too long and will be too disruptive. Real change is disruptive, goes against the culture of the organization, and  can take a long time, but organizations make their challenges greater by engaging in superficial change that only engenders cynicism. 

What organizations usually lack to guide them is a fast enough feedback loop that enables them to try new ideas and run experiments. In addition, they try to avoid failure at all cost, which prevents them from running risky experiments. Because they cannot run small and fast experiments, they overinvest in ideas that turn out to miss the mark, making them even more reluctant to run experiments.  

There is no way to be agile with huge products that require many hundreds or thousands of people to develop because such large organizations cannot run fast feedback loops. The solution is not to create a kind of pseudo-agility that makes the traditional organization a little more agile; the solution is to break down the traditional organization so that it can be dramatically more agile.

The first step on this journey is to break down complex products into simpler products, aligned with customer outcomes and market segments, connected to one another through a common inner-sourced platform. Empowered self-managing cross-functional teams align around these products, supported by a variety of techniques that help them remove cross-team dependencies and reduce waste.

About the Authors

Rate this Article

Adoption
Style

BT