BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Practical Applications of Complexity Theory in Software and Digital Products Development

Practical Applications of Complexity Theory in Software and Digital Products Development

Bookmarks

Key Takeaways

  • There are several practices inspired by Complexity, to be employed every day in software and digital products development.
  • The landscape of such Complexity-inspired practices has three major regions: the teams, the work to do, and the whole organisation.
  • These practices provide a new means to gradually introduce our co-workers, our teams and our organisation to Complexity. They exemplify a new approach that consists of acting our way into the new Complexity thinking.
  • This new approach shifts up-side-down the centre of gravity of the conversation around Complexity, from the theory to the practice. It makes practice prominent. And it wants to engage hands-on practitioners, problem-solvers and tinkerers.
  • All these practices are based on social complexity and anthro-complexity. Therefore, they fully recognise the unique qualities of the human element, and that we are not algorithms, cellular automata, nor hives.

Complexity theory was introduced to the Agile community in 2004 (Joseph Pelrine with self-organisation and social complexity; Dave Snowden with Cynefin; followed by many others), with the intention of explaining why Agile works, and to understand many things about Agile; to pitch different ways of looking at and understanding how modern organisations excel. Complexity theory, together with Agile and Lean, is indeed one of the three pillars of today’s modern ways of working. 

17 years later, when we observe teams at work in their day to day activities, we still see a lot of opportunities to introduce practical applications of Complexity.

What if we start a new conversation about Complexity, also engaging a completely different crowd- the hands-on practitioners, the problem solvers, the tinkerers? What if we approach that conversation in another way? Let’s begin to do that, now, guided by two new radical ideas.

This article is inspired by the book Living Complexity by Luca Minudel.  

What are the two radical new ideas for Complexity?

The first radical idea has to do with the theory and practice of Complexity. The second radical idea has to do with the human element in Complexity theory. 

Let’s start with the first one. Most of the literature on Complexity and most of the conversations revolving around Complexity are theoretical. This is true and has been true in the last 17 years, also in the software development community, in the products development community, and more in general in the broader Lean and Agile community. 

When you look into real teams and organisations, here and there you will find some individual who is passionate about Complexity, who knows the theory, and who is using it to interpret, understand, and make sense of the events happening around her/him and reacting in more effective ways. Complexity gives her/him an edge. But such a presence of Complexity thinking is confined.

The first new radical idea is to shift up-side-down the centre of gravity of the conversation around Complexity; to make the practical applications of Complexity theory prominent.

This means to talk about and discuss Complexity from a practical point of view; to engage hands-on practitioners, problem solvers and tinkerers in the conversation; to act our way into the new Complexity thinking, instead of waiting for the new thinking to materialise itself, dwelling on the theory with only limited action, or philosophising about insoluble problems while forgetting about today’s opportunities to act.

How would “acting into the new thinking” look as a way of making Complexity practical?

The preferred approach to learning new ideas for most individuals is to start with practical examples and actual hands-on practice. Practical applications are the most effective way to introduce the basic vocabulary and concepts of Complexity to our co-workers, teammates, the whole team, and the rest of the organisation.

Being explicit about the context is also important, in order to move from theory to practice. Otherwise, we remain in the abstract and have a lot of implicit assumptions. For that reason I’ve chosen a context for the practices I’m focusing on. It is modern software and digital products development.

The next step is to dig into the practical applications, into the specific practices inspired by Complexity theory. And that starts with understanding what a practice inspired by Complexity theory is. When I ask someone to make an example of a practice inspired by Complexity theory, the answer may be something like Cynefin, the popular framework developed by Dave Snowden and others. In the same way, Extreme Programming and Scrum are Agile frameworks, but not practices. When I ask again, the answer may be something like Probing or Ritual dissent. These are techniques just like Dot-voting which is a facilitation technique that can be used during a Retrospective. Rather, a practice is a set of activities that can be used by a group of people (a team, a few teams, a department, etc.) to fulfil a specific need they have. So each practice has a specific purpose. It has a sequence of steps and activities, a certain workflow. It has inputs and produces a set of outputs and outcomes.

Can you give an example of a practice?

I can give you a quick overview of the Basic Model, a practice inspired by the work of Joseph Pelrine on coaching teams and self-organisation. Pelrine is a pioneer and top expert on Agile methods, who applies leading-edge techniques from social complexity and psychology.

When you are forming a new team- an Agile team most likely, or when you are rebooting an existing team- it is the perfect time to ensure that the prerequisites to self-organisation are in place, so that the potentiality of self-organisation can be fully exploited. Then the team needs a way to try to orient self-organisation toward positive and desirable outcomes. Otherwise, self-organisation could go in any direction, desirable and not.

Self-organisation is a fundamental dynamic of complex systems. This is how this practice is related to Complexity theory.
The Basic Model starts with a check of the prerequisites to self-organisation, briefly listed here:

  • Critical mass: no emergent behaviour occurs from a group of people that is too small. Three members is the bare minimum.
  • Diversity and dissent: homogeneous groups are easy victims of Groupthink and blind spots.
  • Environment: small enough for team members to not avoid each other, and big enough to leave room for manoeuvre.
  • Letting people do it: to interact, collaborate, negotiate meaning and intentions, be creative, act.

Figure 1

The next step is to tweak a few control knobs to try to orient self-organisation toward beneficial outcomes.
The Basic Model introduces this concept of “control knob” and describes three of them. They are briefly listed here:

  • Team Size: has an impact on the receptivity to new ideas, and the time to build consensus.
  • Team’s Boundaries: changing who is in the team or out, and who has an impact on the team dynamic.
  • Roles: activate identities that influence the behaviour (take the SM and PO roles in Scrum as an example)

Figure 2

Some tips and stories following the description of the practice contribute to bringing the practice to life.

One of the tips is about how to use the control knobs. Because of the nature of self-organisation, we don’t and can’t know in advance what patterns will emerge after we tweak a control knob, or what secondary effects or unintended consequences there may be.

So the approach is to tweak the control knob that seems the most suitable for the circumstances, then wait for the effects to unfold and emerge. If they are beneficial, one can stop or even try to amplify that change. Or if they are detrimental, one can reverse the changes and try something different. So the approach for the interventions on the control knobs is trial and error. The intervention cannot be designed upfront given the desired outcome, because the emergence of population-wide patterns of behaviours and actions in self-organisation defy deterministic cause-effect law.

The story below is an account of a personal experience:

From personal experience, working with traditional medium and large organisations, at times I’ve noticed some implicit assumptions and established practices that tend to promote a way of doing things that hinder and frustrate self-organisation, instead of enabling and supporting it. For example, a project-centric approach that disbands teams at the end of each project or often; it frustrates the efforts made to create an effective team. Division of labour and functional silos with individual performance reviews hinder collaboration. A structural separation between planning and execution, thinking and doing, creates a cramped environment with not enough room for manoeuvre.
 

This story suggests that when checking the prerequisites for self-organisation, it is worth watching out for those implicit assumptions and established practices mentioned above and consider the impact they are having on self-organisation. 

Here is one tip.
Don’t tweak many control knobs one after the other like a mad scientist trying every possible experiment. The individuals and the system have memory, disposition, and propensity that are altered by every experiment, even after the experiment is reversed. Instead, start with observing the forces at work, develop some situational awareness around the self-organising system, and intervene only when a change/improvement is required.

This description of this Basic model practice that I’ve given so far is itself an example of how any practice inspired by Complexity theory can be described with five sections: overview, purpose, relation to complexity, description, and practical tips & stories.

Figure 3

What about other practices?

Another practice, for example, is Estimating Complexity by Liz Keogh.
It can be used to estimate the degree of complexity of the work to do, where the work to do could be a problem to solve, a feature to implement, an Epic or a small set of User Stories.
In a nutshell, the practice describes a Complexity estimation scale that follows from the question, “Who in the world has done this before?” 

Figure 4

I’ve mentioned before the Basic Model by Joseph Pelrine, and now with Estimating Complexity by Liz Keogh we have two practices. There are more.

Since 2004, when I began my journey into Complexity, I’ve struggled to find practical applications of Complexity theory in the context of software and digital products development. Since then I’ve kept a list with any practice I could find and any model that could have a practical application; I have designed and experimented with a few practices myself, and I’ve asked other practitioners about their own practical applications of Complexity, up to the point when I’ve found myself with a list of several practices inspired by Complexity theory. I’ve selected the most relevant, I’ve grouped them searching for commonalities, I’ve sorted them from smallest to biggest as a potential order of adoption. The result is more than a digest of practices. One may call it a catalogue of practices whose structure gives a hint of the landscape of all the possible practices inspired by Complexity, in the context of software and digital products development.

Lists and catalogues are attempts to make infinity comprehensible, to create order out of chaos.

The list is the origin of culture - Umberto Eco

The landscape hinted by the catalogue may suggest where to search for other practices, and ideas for novel practical applications you can develop and experiment with. Below is a list of a few practices from the catalogue organised in groups:

  • Basic model (by Joseph Pelrine)
  • Heat model (by Joseph Pelrine)
  • Flow model (by Joseph Pelrine)
  • ABIDE model (by Dave Snowden)

All the practices above are about self-organisation. More specifically, about attending to and orienting human self-organisation toward desirable and beneficial outcomes. Otherwise, anything can happen when self-organisation is left to itself. These practices are centred around the idea that we are complex. Where “we” means individuals and teams.

The practices from the list above range from smallest to biggest. The Flow model extends the Heat model, that extends the Basic model. A Complexity practitioner could start with introducing to a team the smallest practice, build on the success, and gradually introduce bigger and more advanced practices.

  • Sensing Complexity (by Luca Minudel)
  • Estimating Complexity (by Liz Keogh)
  • Complexity Estimation for a delivery initiative (by Luca Minudel)
  • Four points method (by Dave Snowden)

Above is another list of practices from the catalogue. They are about assessing the degree of complexity of the work a team has to do (a problem to solve, a feature to implement, an Epic, a set of User Stories, or a whole delivery initiative). They are centred around the idea that the work we do may be complex.

People who are used to dealing with a lot of simple problems tend to see all problems as simple. People used to dealing with very Complex problems tend to see all problems as Complex. This is why these practices to access the degree of complexity, beyond individual prejudices and biases, are extremely useful.

The practices from the list above also range from smallest to biggest, where the smallest practice can be the first to be introduced to a team, before targeting bigger and more advanced practices.  

  • RUDE estimation (by Dean Latchana)
  • Cynefin for decision-making (by Dave Snowden)
  • C2 Approach Space (by David Alberts)
  • Estimates Accuracy with the Cone of uncertainty (by Luca Minudel)

The practices above are about adapting to the degree of complexity of the work at hand and the circumstances. These practices drive the adaptation of:

  • Estimates
  • Decision-making approach
  • Team settings and ways of working
  • Investment & delivery strategy

Most organisations tend to use an almost static playbook for the way they do things, for any circumstances and all the work regardless of its degree of complexity. This is why these practices are particularly useful.

  • Red Team technique (by Dean Latchana)
  • Culture affinity assessment (by Luca Minudel)
  • Lean Inception (by ThoughtWorks)

These last three practices are about collaborating in the presence of Complexity. They are centred around the idea that the whole organisation may be complex. 

And they are based on co-creation, a collaboration pattern that is particularly effective when dealing with Complexity.

When one looks at the practices in this group, one may recall other Agile practices that could potentially be added to the list. This is because many Agile practices share the same foundations of Human Complexity, namely co-creation, co-evolution, and emergence/simplicity. More generally, these practices that look beyond the team make us reflect on the implications of Complexity on the broader organisation, the way we think about modern organisations, and the way we think about how modern organisations work.

To recap the landscape emerged so far, the practices inspired by Complexity theory in the context of software and digital products development include the three regions depicted below (see Figure 5). 

Figure 5

Zooming in, there are many ways to organise and group and connect the individual practices. I’ve pencilled below one possible visual map and have invited others to draft their own. 

Figure 6

Carlo Volpi accepted the challenge and drafted his own map. His map contains four levels that include his personal notes and comments for every practice. Below is a two-levels version of his map.

Figure 7 (Click to zoom)

What’s the second new radical idea for Complexity?

A large part of the literature on Complexity, and several of the examples used to advocate or explain Complexity theory for teams and organisations, are based on complex systems with agents that are not people. Instead, the agents are algorithms as in cellular automata, in Boids, and the likes. Or molecules as in chemistry models, or cells as in biology models. Or plants, insects, or animals.

But people are not hives nor algorithms. We have notable characteristics that are not found elsewhere.

Figure 8

 

The second new radical idea is to focus on the unique qualities of human agents and human complex systems. We are not hives nor algorithms.

While this may seem to be stating the obvious, a large extent of management literature covering Complexity makes conclusions based on complex systems where the agents are algorithms, cells, or insects.

These are systems where all the agents are identical and easily replaceable, and whose behaviour to some degree can be simulated, preserving the illusion of control, and the possibility to plan upfront actions that will lead to the emergence of the desired population-wide patterns of behaviour and action.

Focusing explicitly on human complex systems with human agents may prevent the mistake of employing the language of Complexity while referring back to previous methods, practices and way of thinking; those that work for the complicated world of the assembly-line and the industrial mass production, but not for the whitewater of the knowledge work, the hyper-connected social and technological ecosystem, and the information overload we live in. 

The branches of Complexity that study this are, for example, social complexity and anthro-complexity. More generally, I refer to it as human complexity.

This radical idea of focusing on Human Complexity is not only appropriate and correct when we look at teams and organisations, but it is also powerful, fascinating, and in a sense even romantic.

Stuart Alan Kauffman, complex systems researcher who studies the origin of life on earth, says that:

We are agents who alter the unfolding of the universe. 

Norbert Elias, a sociologist who used the concepts of self-organisation and emergence in his explanation of social evolution and his theory of civilizing/decivilizing processes, says that:

It is this order of interweaving human impulses and strivings, this social order, which determines the course of historical change.

In the world we live in, in the work we do and in modern organisations, Complexity can be seen as a threat, as well as opportunity. But most of all it is an origin of life, and the source of the most precious ingredient: human creativity. Ralph Stacey tells us that disorder is the material from which life and creativity are built.

How does human Complexity fit in with the practical applications of Complexity theory?

The realisation that people are not hives nor algorithms has huge practical implications.

The agents in a human complex system, unlike non-human complex systems’ agents, are all different from each other; they are heterogeneous, and each one of us is unique.

We, human beings, have free-will, spontaneity, and agency. We are driven not only by instincts, pain or pleasure, needs or desires, but also by beliefs, values, principles, sentiments, emotions, intuition, irrationality, and much more.
And we are intelligent; we have memory and we learn. In addition, we can co-create new knowledge and share it with others.

These are essential qualities that play a fundamental role in knowledge work and modern organisations.

Figure 9

Take a look at the prerequisites to self-organisation in the Basic model previously described, which are necessary for all the four practices based on the self-organisation mentioned before:

  • Basic model (by Joseph Pelrine)
  • Heat model (by Joseph Pelrine)
  • Flow model (by Joseph Pelrine)
  • ABIDE model (by Dave Snowden)

You will notice that among the prerequisites, that diversity and dissent are related to the uniqueness and heterogeneity of the human element, as well as letting people do it that is related to the agency as well as to the intelligence of the human element.

This is really something when you think that in several modern organisations there are still many established practices, implicit assumptions, and habits aimed at limiting variations, constraining creativity, and ensuring compliance. These are rooted in the Industrial Age and the assembly-line.

The approach to orient self-organisation described in the Basic model and used by all other practices based on self-organisation is also based specifically on the human element, for example, the social element of co-creating and sharing knowledge. 
The contrast with other approaches derived by non-human self-organisation (as an example cellular automata and Boids) is stark. One of such approaches consists of designing and prescribing a small number of simple rules in order to create the desired population-wide patterns, as if we human beings are programmable robots.

The human element is also the pivot point of all four practices mentioned before, to assess the degree of complexity of the work a team has to do:

  • Sensing Complexity (by Luca Minudel)
  • Estimating Complexity (by Liz Keogh)
  • Complexity Estimation for a delivery initiative (by Luca Minudel)
  • Four points method (by Dave Snowden)

The result of the assessment reflects the ability of the team members to make sense collectively of the work to be done, what they collectively know about the domain and the technology involved, and their ability to work together.

All the practices mentioned before are based on co-creation which is a fundamental pillar of Human Complexity:

  • Red Team technique (by Dean Latchana)
  • Culture affinity assessment (by Luca Minudel)
  • Lean Inception (by ThoughtWorks) 

The way that Alasdair MacIntyre put it is as follows: << Man is … essentially a story-telling animal. That means I can only answer the question “what am I to do?” If I can answer the prior question of “what story or stories do I find myself a part?” >>

Employees, consultants, partners, suppliers, and their social networks of relationships and local interactions, are potential sources of Complexity. Similarly, hyper-connected and tech-savvy constantly interacting customers are too. Those same networks of human beings who are more susceptible to Complexity are at the same time better suited and equipped to cope with and exploit Complexity.

It takes a network to compete with a network  - Stanley Allen McChrystal

What Complexity-inspired practices could fit into a retrospective?

The first and the smallest among the practices to assess the degree of complexity of the work at hand, Sensing Complexity (by Luca Minudel), has its place in a retrospective.

Sensing Complexity uses two aspects of the human social experience, agreement and certainty, to assess the degree of complexity of the work the team has at hand.

Imagine a situation during a team retrospective where the team members are inspecting the unforeseen reasons for a work item not being successfully completed on time. The conversation, instead of converging toward a solution, seems to go on without finding a good conclusion. There is a lack of consensus on the reasons for the problem faced, and a lack of certainty on what to do to solve it. The facilitator realises that this dynamic may be caused by the degree of complexity of the work item the team is discussing, instead of a team’s dynamic problem.

The facilitator also realises that this is an opportunity to introduce the Sensing Complexity practice, and the basic vocabulary and basic concepts of Complexity. So she/he asks each team member to dot-vote her/his own level of agreement and certainty in the current conversation (see Figure 10 below), to build awareness in the team of the current situation.

Figure 10

If most of the dot-votes reveal a low level of certainty and agreement (three or less) and so most of the dot-votes are in the upper right half of the square, then the facilitator can explain this suggesting that the problem at hand (the work item being discussed) is likely to have a high degree of complexity. The facilitator then continues explaining that the high degree of complexity may be caused by limited, ambiguous or fragmented information available to the team on the work item; or because no one has done this before (solving the problem or implementing a solution, or using the required technology, etc.) or the team has no access to relevant experts; or because other dependencies are causing ripple effects, etc.

The facilitator may suggest the team to consider different approaches to tackle the work item, for example, running a few spikes in parallel, adopting a more gradual and incremental approach to the problem, working more collaboratively with other team members to access a broader skill set and experience (as with pair or ensemble programming), or adopting specific practices for tackling the ripple effect of the dependencies (as using the Mikado method for a code refactoring). These are generic examples the facilitator can share, letting the team and its members explore and discover specific ideas that fit the actual context.

Doing so the facilitator has introduced the team to the concept of Complexity, the idea that the work to be done may have different degrees of complexity, and that different approaches may be required for different degrees of complexity.

The facilitator has also suggested a possible way for team members to perceive Complexity using two common aspects of human social experience, agreement and certainty. And this may be a stepping stone to introducing more advanced concepts and practices to estimate the degree of complexity of the work at hand.

The other practices to assess the degree of complexity of the work at hand mentioned before have their natural place as part of any form of Agile release planning, or Sprint Planning, or Iteration Planning, while the team’s retrospective remains the perfect place to raise perceived changes in the degree of complexity of the work at hand and to schedule a re-assessment.

All the practices based on self-organisation mentioned before have a place in the team’s retrospective too; whenever during a retrospective a team identifies the need or the benefit of trying to change the team dynamic, or to try to orient differently the emerging patterns of behaviour and action. The action to take could be suggested by one of those self-organisation practices and the related 20-odd control knobs that can be tweaked. The team’s retrospective is also a perfect place to evaluate the effects of previous tweaks, and to consider if any follow-up action is required.

Conclusions

If, starting from tomorrow, you want to introduce the concept of Complexity and its new way of thinking to your co-workers, teammates, other teams, and the whole organisation, then the most effective way is to act your way into the new Complexity thinking. You can do so using several practices inspired by Complexity to be employed every day in software and digital products development.

This means introducing small Complexity inspired practices, gradually, starting from the small ones that also introduce the basic vocabulary and concepts of Complexity, then moving on to introduce the more advanced ones. This also means extending the conversations on Complexity to include hands-on practitioners, problem solvers and tinkerers, and shifting the centre of gravity of the conversation from theory to practice.

The other important aspect is to focus on practices from Human Complexity (e.g, social complexity and anthro-complexity) that take into account the human element, as we are not hives nor algorithms, and what works for them doesn't necessarily work for us.

The overall landscape of the Complexity-inspired practices we have introduced here has three major regions: the teams, the work to do, and the whole organisation.

Here we provided a few examples of such practises, and we have listed several other practices.

A full description and elaboration of all the practices mentioned here, together with more tips and stories, are available in the book Living Complexity by Luca Minudel. 

There is also a new federated wiki available for Complexity practitioners, to comment and discuss the practices, as well as to suggest new practices.

About the Author

Luca Minudel is a Lean-Agile coach & trainer, transformation lead, author and speaker with over 20 years of experience in professional software delivery and digital product development, most of them with Lean and Agile. He is passionate about agility, lean, complexity science, and co-creation. Minudel contributed to the adoption of lean and agile practices by Ferrari's F1 racing team. For ThoughtWorks, he delivered training, coaching, assessments and organisational transformations in top-tier organisations in the UK, Europe and the United States. He worked as head of Agility, Agile transformation lead, Lean/Agile practice lead, and as Lean/Agile coach in companies such as HSBC, Lloyds, LexisNexis. Minudel is the founder and CEO at SmHarter.com, a company that helps organisations turn their way of working into a competitive advantage. TwitterLinked-InBlog.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Anthro-complexity, acknowledgement of the source (addendum from the article's author)

    by Luca Minudel,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Anthro-complexity has been pioneered by Dave Snowden and The Cynefin Centre via its applications. Anthro-complexity is a radical new discipline which studies the human aspects of complex adaptive systems and argues that human beings are radically different from others.

    The 3Is of Anthro-complexity mentioned in Figure 9: Identity, Intentionality, and Intelligence have also been defined by Dave Snowden.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT