Foundations of Self-Organization

The idea of self-organizing teams has been called the secret sauce of agile software development. This paper is an attempt to systemize how to think about and how to develop healthy self-organization.

Introduction

I first started to think about self-organization in software development teams in 2011. At that time there was not a whole lot written about it; Except that you should have it if you aspire to be agile. There was hardly any advice on how (as a manager/coach/leader) to foster and nurture self-organization and most advice was along the lines of:

• Don’t assign roles
• Don’t assign leadership
• Don’t assign tasks
• Don’t say how

That is a lot of don’ts and while it certainly is useful to be aware of anti-patterns the question of what you actually can do to support self-organization was largely left unanswered.

Today you can more easily find advice on self-organization of agile teams on the internet, for example here and here but the subject is still under-described and underdeveloped as I see it. The model that I outline in this post is not completely unique, but I would like to think that it is a bit more clear and focused compared to some other descriptions. My hope is that this write-up can act as inspiration and perhaps as a starting point for discussing how to best leverage self-organization in your organization.

What and Why

There are many variations of how to define a team as opposed to a group of people. A succinct but still useful definition is:

A team is a group of people with complementary talents and skills aligned to a common purpose.

This is a generic definition and it is in no way specific to agile or software development teams.

Sometimes there is a distinction made between a self-organizing, self-directed, and self-managed teams. To further complicate things different people define these terms differently^[1]. Making this distinction is not so important in the context of this article. However, the assumption I have had when writing the article is that the team we are considering exists in a context where there are several teams with a common or at least related purpose and also that there is some kind of management structure around the teams. I.e. the teams are not assembled spontaneously and they have some level of interdependency.

So why then is it, from a business and management perspective, a good idea to have self-organizing teams? While it certainly is nice to be perceived as a modern organization with sound values and empowered employees the real reason is much more down to earth; Self-organization is a powerful management strategy which will lead to better results because:

• Assigning end-to-end ownership to a team for delivering something of significant value to the business will make people more motivated to excel which in the end will result in higher quality in deliverables.
• Local decision making means quicker and also better decision making which in the end will lead to quicker deliveries that are more fit for purpose.
• Since there are no (or at least fewer) hand-offs when teams get end-to-end responsibility, you will get more done more quickly as you get to focus on one thing at the time as opposed to switching between different tasks all the time.

In addition to an improved outcome in pure business terms there is also a set of characteristics, or values, that a well-working self-organizing agile team should demonstrate, for example:

• Commitment
• Courage
• Accountability
• Honesty
• Transparency
• Trust
• Openness
• Authenticity
• Respect
• Communication
• Feedback
• Simplicity

In some texts it is implied that the way to foster self-organization is to make sure that these characteristics are present in the team. However, I would like to argue that these characteristics are more of a proof of well working self-organization rather than a recipe for getting there. Instead, I would like to argue that there is a set of organizational preconditions that must be in place to make self-organization possible. Then, given these prerequisites, we also need to be mindful of that healthy self-organization is heavily dependent on the people in the team. The picture below tries to illustrate this, i.e., first you need to have a baseline of organizational assets in place which I call foundations. On top of that you then need to make sure that you have healthy teams and well-rounded individuals in those teams. Only then can we achieve healthy self-organization with real team values and behaviors like the ones listed above.

Let us look at the different layers of the model in more detail, starting with the foundations layer.

Foundations

In the foundations layer we find the organizational infrastructure as follows^[2]:

• Goal and Utility
• Knowledge and Learning
• Communication and Feedback
• Way-of-working and Decision Making
• High standards and Expectations

Let us briefly discuss each of these points.

Goal and Utility

To get any healthy self-organization, you must have a significant goal to strive for and know how to measure the utility of the team's outputs and services in an ongoing fashion.

In economics the most simple way to define utility is the amount of money someone is willing to pay for some goods or service. Using cost-of-delay^[3] as the measurement of value has become quite popular in agile in the last couple of years and it is probably the best proxy for utility we have. Put in another way, a team must have a way of determining how effective it is in strict economic terms. Measuring value/utility is not without challenges:

• It is difficult to reason about cost of delay/value at the level of individual user stories. You typically need to move up at least one level.
• Even when you move up one step it is not straight forward to estimate the cost of delay/value in a reliable way.
• It is even more unclear how to calculate cost of delay/value for component teams and infrastructure teams.

There are a few pointers on blackswanfarming.com that can help you get started thinking about value^[4] and urgency to calculate cost of delay^[5] but there is still significant work involved in making it relevant in your specific business context.

In addition to having an understanding of utility it is also important for the team to have a significant high-level goal to strive for. A good high-level goal should include:

• Intent and end-state – This is focused on the “why” of the team's work and the change to be accomplished. In software development this is typically expressed in terms of opportunity enablement for customers and the business.
• Main effort – There will be many details that must be dealt with to ship a complete product release, or to make a significant change. Defining what the main effort of an endeavour is typically phrased as a list of key features that are intended to solve the problem at hand.

This is similar to how mission orders are structured in the US Marines and other military organizations.

Many organizations have at the top level defined goals in terms of economic outcomes: revenue, profitability and so on. However, this kind of goals are completely ineffective as a means to enabling self-organization. For a goal to function as effective guide you need to feel that you (as a team, as an individual) have a significant degree of influence on whether the goal is reached or not. That is not the case if you define the goal in terms of top-level KPI’s. The goal need to be something you care about and find exciting and challenging.

One of the best examples that I have seen of a goal that really made a difference is from almost 20 years back; A time when software was still shipped in cardboard boxes. I was then working for Rational Software with tools for software development and one of Rational’s products was the then quite popular UML modelling-tool Rational Rose. Back then Microsoft was the premier tech super-unicorn that everyone had to be mindful of. Anyway, Rational made a deal with Microsoft to develop a variant of Rational Rose called Microsoft Visual Modeler that was to be shipped as part of Visual Basic 5.0 / Visual Studio 97. The schedule was pretty aggressive to get the product done and the team that worked on it was a result of a recent merger and was distributed across the US and Sweden. Not an easy thing to succeed with. The high level goal that made the team come together and work as one was simply expressed as:

Ship in the box with Visual Studio 97

This was a great goal as it both was something exciting and desirable, as well as challenging for the development team. It was also fully linked to the business goal for Rational which was to get maximum exposure to the enterprise developers using Visual Basic and having a compelling up-sell to offer. In the end it turned out that it was not the right decision to ship the product in the box on the Microsoft CDs but it was better to distribute Visual Modeler as a download. It was however a great goal in terms of getting the product done^[6].

There is much more to be said about how to work with goals and measurement (of utility) and I hope to get the time to expand on that in a future post.

Knowledge and Learning

Sometimes people believe that to be able to self-organize a team must have all the necessary knowledge in the team from the get-go. In reality it is enough to have most of the required knowledge. What remains can be learned as we go. Consequently every team must have an understanding of what:

• A. Things we know.
• B. Things we are aware of that we don’t know, but we need to know them to be successful.

In addition to that there is also the ugly category of knowledge that can be classified as:

• C. Things we are unaware of that we don’t know, but we need to know them to be successful.

So, to be successful we need a way to learn things in category B and C and we also need a way to become aware of things in category C. The team needs help, time, and perhaps funding to achieve this. Without going into too much detail, it is essentially the responsibility of the organizational system around the team to provide the proper support. Examples of this include:

• Engagement of external stakeholders/business people that can educate the team as required in the business domain for which they are creating a solution.
• Technical supervision to make sure that the team is aware of standards, design patterns, and design intent of any existing solution that the team will touch. It is important that this kind of supervision is done in a way that does not come across as authoritative or that it delays the work of the team. Instead of setting up formal approvals, people tasked with this kind of supervision should practice go-see for example by attending sprint reviews, sitting in on design workshops and similar. This way you can educate the team in context and it will become more and more self-sufficient as time passes.
• Time, funds, and moral support from line management to make sure that the team takes the time to learn what is necessary. Examples here include dedicated learning days across the organization and personal budgets for training.

Learning is of course related to feedback which is discussed next.

Communication and Feedback

A self-organizing team will continuously adapt what they build, intend to build and how they go about it. To do this the team must get feedback on its work. What you need feedback on is:

• Are we building the right thing, i.e., are we solving the problems that really needs to be solved or are we working based on a misconception of the real needs? We can think of this as feedback from stakeholders/users of the system we are building.
• Are we working in the best way, i.e., is there anything we can change to become even better as a team? We can think of this as process feedback.
• Are we building the thing right , i.e., does the solution we are building have the right quality attributes and is the level of faults tolerable? We can think of this as technical feedback.

Not by coincidence, feedback is at the heart of agile methodologies. Agile teams typically have regular demos/reviews with stakeholders and business people to get feedback on the solution they are building; they hold retrospectives to assess their way-of-working and they employ technical practices such as automated testing and continuous integration to get a constant feed of technical feedback.

Communication within a single team can certainly be a challenge in itself but in this model this falls into the People layer discussed later on. However, in multi-team scenarios you are well served to consciously look at the communication between teams. For meaningful communication to take place you must both have mechanisms for communication and a shared ontology^[7]. If you only have a few co-located teams and the teams have a shared history this will for the most part take care of itself. If you on the other hand have many teams that don’t have a shared history, teams may need support to establish the mechanisms of communication and a shared ontology. This will be further amplified if you have teams that are distributed and/or belong to different organizations as would be if you are working with an outsourcing partner.

In practice this means that you will need to establish meeting schedules, make sure that there is adequate space and technology to run, e.g., distributed meetings. You also need to establish a culture where people don’t hesitate to reach out to other teams for help or to discuss things of common interest. To be able to do that there must be some kind of directory of what teams exist, what they work with, and how you can contact them.

If your organization is transitioning from a traditional hierarchical organization this will require people to learn new behaviours. When you have self-organizing teams, coordination and requests are handled peer-to-peer rather than going up and down the project or line management hierarchy. In my experience you explicitly need to tell people in teams that it is their responsibility to reach out to others. In a large organization this means that you sometimes need to call someone that you don’t know and talk to them. For some team members this will be uncomfortable at first but my experience is that once you have broken the seal everyone appreciates the benefits of direct communication. An anti-pattern that you sometimes see is that the scrum master/agile coach handles all external communication on the behalf of the team. In my mind this is an unfortunate step back towards a traditional project management role.

A shared ontology should be documented somehow. For example, one possibility is to capture the most important concepts in the business domain in a domain model^[8]. But it is easy to over-document and if a model gets too large or too detailed it will quickly become obsolete and/or hard to understand, so stay lean. High level architectural descriptions and guidelines can serve a similar purpose in the technical domain. The purpose of these kind of descriptions is to raise awareness about what the interesting things to talk about are rather than to specify things in detail.

Way-of-working and Decision making

Any intellectual endeavour, which software development is, requires you to make decisions. Sometimes the decisions are simple to make, sometimes they are complex. Sometimes the impact of a decision is of a local nature and sometimes it is of a global nature. In a hierarchical model with clearly assigned responsibilities it is (at least in theory) easy to understand who does what and where a certain class of decisions should be made. When self-organizing teams come into the picture it is not so obvious anymore who makes the decisions.

Further, to work effectively as a team you need to have a level of agreement on how the work is done, i.e., a way-of-working. A self-organizing team has a level of freedom to define this way-of-working but in a larger organizations there are also things that must be agreed across multiple teams or even across the organization as a whole. Essentially an agreed way-of-working should help collaboration among individuals and teams and make sure that timely decisions of high quality are made at the right level in the organization.

It is also so that there are different kinds of decisions to be made. Largely there are three different domains that decisions concerning software development fall in:

• The business domain
• The technical domain
• The way-of-working domain

For each domain it is useful to come up with a classification of at what level different types of decision should be made. The goal should be that decisions with small economic and organizational impact can be made locally while decisions with large organizational impact or large economic consequences are done in a concerted fashion while still avoiding long lead times and decision stalemate.

Examples of different levels of decision making in an agile organization with multiple teams could be:

• Individual
• Team
• Program (a group of teams working on the same solution)
• Organization

Considering these two dimensions we can form a matrix, like below, helping to understand where a certain decision should be made.

The content of this matrix will be different for different organizations but it should be able to answer questions like:

• Who decides what acceptable working hours are and what the policy for working from home is?
• Who decides what technologies to use and to not use?
• How much budget can each level decide on independently?
• What equipment can be made available to teams and individuals?
• What tools are available?
• What is the definition-of-done?
• What is in scope and what is out of scope for the solution being worked on?
• What is the release schedule?
• How do we test?
• How do we do code/design reviews?

Once it is agreed where a certain class of decisions is made it is of course important to make sure that you are nimble in making decisions regardless of what level the decisions are made as the opposite will slow things down and hinder progress. On the individual and the team levels this is not much of a problem as decisions can be made pretty much on demand. On the program level and the organization level you typically need to have a formal decision forum with scheduled meetings. On the program level such a forum typically will have representation from the teams and program management. On the organizational level it will typically be line management that makes the decisions but programs and teams should of course be consulted as appropriate. It is better to have short and frequent meetings rather than long and less frequent to make things flow smoothly. Program level decision bodies should preferably convene daily while organization level decision bodies can convene somewhat less frequently.

Decision making is a subject well worthy a separate discussion. I hope to be able to get back to that before too long.

High standards and Expectations

Self-organization does not mean that there are no performance requirements. Similarly as having a clear and challenging goal it is equally important to establish a culture of excellence, or perhaps a culture of aspiring for excellence. So how can you do this in a team? Some years ago I attended a talk by the coach for one of Sweden’s leading football clubs^[9] (they had just won the league that year). The talk touched on how to, in a quite diverse group, create a shared urge for excellence. Some quotes that stuck with me are^[10]:

• “Do simple things and do them every day”, i.e., with high quality
• “Good is always fun but fun is not always good”

As I understand it these were a kind of mottos they had in the team in order to:

• Establish discipline and attention to detail
• Establish a relentless focus on quality and outcomes in everything

I think the effect of this kind of mindset translates well to software development teams where the best teams also are very detail and quality oriented, for example in terms of:

• Code quality
• Adequate automation of e.g., test and build
• Setting challenging but achievable goals for themselves
• Delivering on their commitments
• How they constantly strive to become better

I also believe that, in this respect, the coach has a similar role to play in a software development team as in the football team. i.e., to inspire and challenge the team to strive for excellence and perhaps establish a suitable motto for the team.

Now let us look in more detail at the people layer in this model for self-organization.

People

Despite that we work with technology and despite different techniques for capturing requirements, making estimates, or managing a team's work and so on, it is still people that need to collaborate to make things happen in a team. When considering this it is important to see the team as one entity and all the team members as individuals separate from the team. To get great results from a team you must make sure that you both have motivated individuals and great team dynamics.

Motivated individuals

Something that I have learned the hard way is that motivation is, at least to a very large extent, a personal responsibility.  Some years ago I hung on to a position a bit too long for strict economical reasons rather than having an urge to succeed in my role. To make a long story short; What you personally feel is lack of motivation will be seen as poor attitude by others. That will definitely not increase your chances of being perceived as a valuable contributor or someone to invest in for the future. My take-away from this is that you cannot expect your boss or someone else in the workplace to motivate you. After all you are responsible for your own future. Kind of obvious isn’t it. What an organization can do is to create an environment where individuals can become motivated by themselves.

The figure below shows the three necessary ingredients for what is called intrinsic motivation.

This model for motivation comes from the body of psychological research^[11] that Dan Pink based his popular book Drive^[12] on and the TED talk^[13] on the same subject. Autonomy, Competence, and Relatedness in the figure above is what Dan Pink calls Autonomy, Mastery, and Purpose.

What you absolutely shouldn’t do is to create silly red-tape processes that take away motivation from the individuals. It is important for organizations to understand that it is much easier to demotivate people than to motivate them. So then, what can you do to grow motivation? As stated in the beginning of this section, my belief is that intrinsic motivation is fundamentally a personal responsibility. I.e., individuals need to put themselves in a position where they can become motivated. What organizations need to do is to be aware and supportive of these mechanism and create a culture where intrinsic motivation can grow. A bit more about this in the next section.

Group development

Many books and much research has been made on group development and team dynamics. Some texts are more well-founded in psychological research than others. A well-founded and up to date model is the one suggested by Susan Wheelan^[14] that is shown in the figure below.

When you overlay this model with the model for intrinsic motivation described above one can conclude that an important aspect of the early phases of group development is to find common ground and working agreements that make it possible for all members to become intrinsically motivated by:

• Autonomy - This means having a sufficient level of influence of how you work and what you work with. Dan Pink talks about autonomy as the freedom to select team, task, time, and technique. I.e. who you work with, what you work with, when you do the work and how you go about it. Agile have this built-in to a significant extent in that teams should be allowed to decide on these things themselves as long as their decisions don’t negatively effect others. Autonomy does however not mean complete independence or that you can do as you please without considering the needs of others. Examples of where agile increases autonomy compared to traditional practices include: team members volunteering for tasks rather than being assigned, team members can pairing up with other team members as they like, direct communication with stakeholders, teams deciding themselves on development practices, and more.
• Competence/Mastery - This means that you have a feeling of being competent and are doing a good job. This requires an appropriate level of challenge in the work you do, given your individual knowledge and desire to learn. But this is not enough as the single most important factor for someone to feel competent is positive feedback. I.e. positive feedback fuels motivation while negative feedback reduces it. So make sure that you have mechanisms set up to give teams and individuals significantly more positive than negative feedback. In addition to regular management and stakeholder feedback, look at ways to stimulate peer to feedback with kudos^[15] or similar mechanisms.
• Relatedness/Purpose - This means a strong level of buy-in into the organization’s/team’s goal as discussed under Foundations, especially as it pertains to the why and the desired end state. It also means that you feel related to the people that you work with in that you have shared urge to reach the goal. If the why and the end state are clear and well communicated and people still can’t buy in, then they are probably in the wrong place.

While an agile coach can facilitate and have an influence on some of these things it is ultimately the line management of the organization that is responsible for that the teams and the individuals of the organization function well. My recommendation is simply to talk to teams about these things, maybe as part of an extended retrospective. It doesn’t need to be so complicated but just raising the awareness of these mechanisms in the team by presenting them and having a group discussion will in my experience help to progress things. I would not do this directly with a newly formed team as you will get more value from the discussion if they have at least a month of experience working together. The simple models for group development and individual motivation described here can act as guidance here.

Outcomes

The final layer in the model is about the outcomes that you can hope for provided you have the first two layers sufficiently in place. I believe that there are the three key outcomes of successful self-organization:

• Values
• Balance
• Results

Values

The model described here does not suggest that any desirable agile values will appear automatically without talking about them. Rather it suggests that you cannot expect them to appear if you don’t have the Foundations and People layers in the model in place first.

Agile values typically include the ones listed above, i.e.:

• Commitment
• Courage
• Accountability
• Honesty
• Transparency
• Trust
• Openness
• Authenticity
• Respect
• Communication
• Feedback
• Simplicity

It is the responsibility of the agile coach to make the team aware of these values and why they are important. Unless you are very sure of what you are doing I would however be careful in engaging in explicit interventions or group exercises meant to force development of, for example, trust, honesty, authenticity and so on. The reason to tread carefully is of course that there always is a risk that individuals can get hurt irrevocably when doing this. I believe it is better to spend the energy on making sure that the two lower layers in the model are in place and that the team can successfully deliver value with high quality. Nothing builds team spirit like winning.

Naturally line management also plays a key part when it comes to growing the culture and the values in an organization. If the executive decision makers are perceived as insincere pretty much everything else will be ineffective. I.e., to grow values you need people at the senior level that lead by example.

Balance

Balance refers to that self-organization really means that the team has the authority and the ability to continuously make tradeoffs. Examples of tradeoffs that you need to make in a software development team include the ones listed in the figure below:

You can probably come up with a few more trade-offs a team need to make yourself. This is something I picked up when reading the doctoral thesis by Rashina Hoda^[16].

Results

In the end it is all about results in terms of reaching goals and maximizing utility. As suggested in the beginning self-organization is a powerful management strategy and a healthy self-organization simply have a better chance in delivering the results compared to strictly directed teams.

A Note on Complex Adaptive Systems

Some people have suggested that you can understand self-organizing teams as complex adaptive systems as in complexity theory. I must admit that I too was intrigued by this at first. However after some time I found that this was a dead end which does not help you to understand how a team works. It is of course true that an agile team both might be complex in its interactions and that it is adaptive. However, if we look at what defines a complex adaptive system it has the following properties:

• Complex, dynamic network of interactions and relationships
• Adaptive as behavior changes as a result of experience
• Key principles are:
  • Self-organization
  • Emergence

If we just take these words and map them onto an agile team you can easily come up with a train of thought like this:

1. We want self-organizing teams
2. Groups of people left on their own show emergent behavior
3. Therefore we can understand a self-organizing team as a complex adaptive system

In my mind this is however quite misguided as complex adaptive systems in complexity theory also means:

• The individual agents in a complex adaptive system follow a simple set of rules, it is the system as a whole that is complex, not the individual agents.
• There is a large number of individual agents in a complex adaptive system and the simple interactions of the agents in the system can then self-organize and show emergent behavior in complex (not easily predicted) ways.

This is quite different from your typical agile team where we have a fairly small number of agents (people) and where each individual agent have very complex interactions and behavior, remember it is people we are dealing with, not bees.

To conclude, complex adaptive systems cannot be used as a valid model for self-organizing agile teams and even if they could it would not help us much in understanding how we can support and nurture self-organization in the workplace.

Conclusion

Much of what is discussed in this paper should already be familiar to experienced agile leaders and coaches. Further the paper does not contain much in terms of concrete facilitation techniques or exercises that can be applied directly to foster self-organization. What I hope instead is that you can use this model as a tool to look at your own organization and see if you have the necessary pieces in place to enable healthy self-organization. This is of course just a theory but I hope that you find that it is a good or at least somewhat useful theory^[17].

So here it is in one figure, my Grand unified theory of self-organization:

Even though the model suggests that you, mostly, need to tend to the items toward the bottom before items higher up there not a strict order. In practice you will find that you need to work with these items in parallel and iteratively. This is what the arrow indicates.

About the Author

Svante Lidman has worked with software development and management/coaching of software development teams for more than 25 years in start-ups as well as large international companies such as Microsoft, Ericsson, and Rational Software. Since 2008 Svante has worked as a consultant, mainly helping software development organizations to change towards lean/agile. During 2010-2012 he was involved in initiating and driving one of the largest lean/agile transitions done in Scandinavia.

I would love to hear your feedback on what I have written here. Post any comments below or contact me on twitter (@svante_lidman) or LinkedIn.