Dan Greening on Chaos Theory and Agile Base Patterns

1. Good day, folks. This is Shane Hastie, we are here at Agile 2015 and I am talking with Dan Greening. Dan, welcome! Nice to see you again! Dan, we know each other, but I suspect a fair number of our audience do not so would you very briefly introduce yourself?

I am a computer scientist by training. About 20 years ago I was really interested in complexity theory and optimization and high parallelism and a bunch of other super nerdy things and then I starting being a Start-Up guy. I started three companies, two of which were reasonably successful, one of which was a bust and then, at that time, at that bust, was the last one and I said “This is really interesting. I need to learn from this failed company. What could I have done better?” And I realized that what I could have done better is have been a better product manager, finding the markets for things and adapting my company to make more money in those markets.

After that, I joined a company called Citrix Online and when I did that I had a team, we were in the research department and my team was tasked with some new work. One of my team members said “You know, we should use Scrum. We should have daily stand-ups, we should have all the things in Scrum.” And I said “That is interesting. How do I find out more?” He handed me a book and I said “OK. That sounds like it is going to work way better than anything we did before” and we became the first Scrum team in Citrix Online. We got great success from it, that success was noticed by the people and we started being looked to as guiding people and then ultimately I became head of the Agile program Office for Citrix Online. Since that time I was head of Agile coaching for a while at Skype and I have written a lot of material on the stuff that I have been doing in these areas. One of the things that I did for Citrix online was capitalization, a very challenging thing, and I published an article in InfoQ about that.

More recently, I have been interested in Agile scaling for large organizations because I tend to be working with large organizations. I think that is really interesting. If you can make a large organization more Agile, you might be able to improve its productivity by –let’s say it is less than a small team, like 5%, but now you are talking about 5,000 people with 5% improvement and that is an enormous number and it makes a lot more value for society and it makes a lot more people happy and so I like to think about large organizations. This is what has taken me to where I am now, I suppose.

2. And where you are now, you are looking, one of the things we have chatted about is the concept of Agile leadership patterns and Agile base patterns.

Right. So I started thinking probably about four years ago that Agile was best done top-down. It sounds a little bit wrong-headed to some, I think, but the general idea is that if your CEO or the CTO or someone at a very high level, operates in an Agile way. What I found is that if those people understand Agile and they think “Agile applies to me. I need to make sure that I do things rapidly, iteratively, experimentally”, they start realizing that the people underneath them need to be Agile as well to support that and it is almost a virtuous cycle, right? So now the whole organization has to transform in order to serve that high-level agility.

Jeff Sutherland and I have conversations about this frequently, where we talk about the fact that most of the time, when we do bottom-up Agile, we say that it runs into a barrier, sometimes at a Director level or at VP level, where someone does not really understand Agile or it conflicts with their personal goals. Often times, it gets to that level and then it gets destroyed completely, like someone gets threatened by Agile, it’s got a high level of transparency, it also encourages this idea of learning from failure which means we actually talk about failure, which many companies don’t even want to do. But Jeff and I noticed that when higher level people were Agile that did not happen.

The agility was sustained over a long time. We also noticed something else which was that a company could have an Agile leader and then that Agile leader could leave for a different opportunity or retire or whatever and then the company would hire someone who would say they were Agile, but did not really have agility in their soul, I guess that is the way to say it. Then agility would start stumbling and failing in that organization and it would regress. We have seen several large companies that have gone through multiple waves of what I call “armies of coaches” who come in and help the engineering department to become more Agile because they lost it somewhere along the way. What they do not realize is that it is flowing from the top, right? We spend millions of dollars converting these engineering departments, but we have no support at the top and then, when we remove this armies of coaches, what happens is - guess what? – it regresses over time. So you can do this bottom up strategy, but it is expensive and if you really want to embrace this idea, what you do is you get a big budget, you write a big check and you just resign yourself to writing a big check every year for a bunch of sophisticated coaches to keep your company Agile. That is at least the assessment I have.

So, as I started thinking about agility on the organizational level, because I wanted to give guidance to bigger companies, how do you gain and sustain organizational agility. Then I said “Well, what is agility? What am I talking about here and why do we do it?” Conveniently, I am chaos theorist from a long time ago and I never thought I would ever use this in Agile practice. Chaotic systems are actually mathematical systems. They are even deterministic systems. If you know exactly where the system is starting, let’s say a stock market is an example of a chaotic system – if we know exactly where all the markets are, exactly what the motivations of people are, exactly what the state of the world is and we have a perfect simulator, we can simulate the economy in that stock market and we can predict exactly what the stock market will do 20 years from now – that is the theory behind a chaotic system. But if we are off in our initial state by any amount, the inaccuracies of that increases exponentially over time and so what happens is that we really cannot predict what is going to happen 20 years from now. In fact, we might not be able to predict what is going to happen a year from now. It depends on the rate of chaos. So, the best way to deal with these chaotic systems is something called a “complex adaptive system” where you operate for a little while and then you measure again and you say “How far different from where I thought I would be am I and how could I correct for that?” So you do little corrections along the way, a little correction along the way, a little correction along the way and, guess what? – this is iteration. In Agile practice we are trying to achieve a success measured somehow when we are dealing with a chaotic system. So now we can apply some of the knowledge of chaotic systems to what we do.

I started thinking about Agile practices and I lump a whole bunch of things into Agile. Of course, there is Scrum, there is XP,I think Kanban belongs there. I even think things like Getting Things Done, you know, the personal practice of “How do I organize my priorities and complete projects”. I also think that Lean Start Up is an Agile practice. I think that most of us, if we are in the Agile space, we relate to these things. So, what do these things have in common? That was what I started asking because I wanted to have that commonality and then I wanted to say “Hey! You, executives! You should do this philosophical, conceptual thing to your environment”.

There are five. The first one is that you have to know how to measure progress. So, of course, in most companies, like a public company, we want our stock price to go up. In a private company, we want profitability to go up. The problem with those things is they’re lagging metrics, right? We do some work, we have some improvement and then a year or two years after we do the work we are starting to see the profitability, we are starting to see the stock rise, but it is not Agile. We cannot work in this chaotic system and then say “Hey, you know what? That thing that we tried actually improved our situation” We will not know for a couple of years. So what we need is leading indicators. The first pattern of this Agile based pattern is to measure economic progress, the idea that we can have a leading indicator to measure how we are doing. You might say “Well, that sounds really great theoretically. What does that mean for me and the Scrum team?” In a Scrum team, we are talking about velocity – is one metric. So, as a Scrum team, we want to produce more functionality, each iteration, the metric we can use is velocity. But one of the problems with metrics is that they can be perverse, if you are not careful. You say “OK. Let’s say we want to improve velocity” and that is the only metric we have, we start pushing on that hard, we start generating bugs, we do not care because bugs are not something we measure, we are just going for velocity and you produce lots of buggy code. So, usually, to avoid perversity, you need additional metrics. You need multiple metrics to measure your economic progress. That is the first pattern. So, defect count and velocity and happiness are really interesting things to measure for production in a Scrum team. Lean Start Up has a converse sense of metrics. Now we are dealing with measuring the market. So we are going to measure hit count, we are going to measure retention, we are going to measure how much people are willing to pay for the service we offer and then we measure that over time and iterate over time to get better and better at meeting the market.

The second pattern is a thing called adaptively experiment for improvement. So, if we just do this metric stuff, the world will help us experiment for us because random stuff will happen in the market and we will be adapting to that over time. The problem is that often times, the world is lagging because it is dealing with our lagging indicators and then it comes back and it pushes on us and by the time we realize that something is going wrong, we are essentially dead. This is the Kodak phenomenon, right? Like, we built all this infrastructure around film cameras and now we have digital cameras coming up and they said “Oh, that is interesting. We would better do something about that, but let’s not worry about it for now.” And then they are bankrupt. GM. Same thing. Like “Oh, this Toyota production system sounds good, but we really do not have time to worry about it. We do not have any leading indicators to tell us we are doing better and by the way, we are bankrupt.” So what you need to do is to actively experiment with your system. We do that in Scrum with retrospectives. Retrospectives are great. They are not often done as experiments, but they could be and when they are done as experiments, you get much better results. So let’s figure out what metrics we are going to improve, let’s say by improving our definition of ready. Then, when we say that, let’s hypothesize how much our velocity will improve. We are going to beef up our definition of ready so that we are not going to have any dependencies around our stories, let’s say and our Scrum team should say “Well, our velocity was 30. What should our velocity be after applying that?” They should hypothesize something because they can go back at the end of the sprint and think about “How well did we do? Were we surprised by a much greater improvement?” Let’s say they say 10% so 33 of velocity. My guess, having some experience as a coach, if you get that ready criteria down, you are going to get much higher than 10%, but I bet most teams will think about of this 10. So, that is the second thing – “adaptively experiment for improvement”

The third one gives us agility. We are not Agile yet. The third one is limit work in process. If you think of this chaotic system I described, if your inspection time is too slow, if your adaptation time is too slow or if you creation time is too slow, you are not going to be able to adapt rapidly enough for this chaos that is moving you all over the space. So if you limit work in process, in other words if you ship at the end of every sprint, at least, and if the sprint cycle time is short enough to actually match the chaotic movement that we see in whatever system we are operating in, you can adapt rapidly enough to make that work.

So those three things give us agility: measure economic progress, adaptively experiment for improvement and finally limit work in process.

The problem is that those three are great and you will be Agile, but it will be hard to keep it. That is what we discovered. So if you did just those things, one of the things that we might not do is, you know, in XP and in Scrum, we often tell people there should not be any developers who exclusively do development. There should not be any testers that exclusively do testing. Why is that? We do that because we want elasticity in the team if we have a higher amount of testing that is required by the team, we do not want the team to be constrained in its ability to test. We do not want developers to be developing code that cannot really be tested because that becomes technical debt that sits there and we might not be able to ship it and also leads to other problems.

The fourth pattern is a little bit broader than that because we want to talk about not development, we want to talk about anything that is Agile. That is embrace collective responsibility. Now the idea is that when you join a team, you sign a contract. You say “By joining this team, I agree that I will be personally responsible for the collective output of the team.” That is a pretty strong statement that is saying “You know what, I have looked at the team members in this team, I think I am comfortable working with them. If we have trouble, we are all going to communicate and we are going to figure out how to solve the problem and get this thing out”. So, that also means that you cannot walk into a team and say “Hey, you know what? I am a developer” and then the team has a train wreck, it produces a piece and ships something or is not able to ship something because it is not sufficiently tested. As a developer I cannot say “I am really sorry that happened, but that really was not my responsibility.

I am a developer” and so you cannot so that anymore. You signed a contract that says that you are going to be personally responsible for the collective output. That creates all sorts of elasticity within the team. Now people feel responsible for learning a little bit more about databases, just in case the database guy is gone. Now the team feels more responsible for dealing with learning new things that are likely to be coming in through the pipeline of the backlog, for example. So that is our second thing and what that brings to the table is greater stability for the team. The team is capable of being Agile, even with routine interruptions or someone poaches an important person out of your team, or they go on vacation or whatever. That is great. So that is fourth one. Now you get a little bit of stability.

Now what we also see is that these teams or individuals, their agility is affected by more than themselves. They are often dependent on people outside or entities outside. One of those things is if we have one team and it is consuming something from another team, let’s say the other team is not Agile, now we have a big problem: our agility is dictated by this other team, we have to figure out how to solve that problem. So that pattern is solve systemic problems. The idea behind systemic problems is we are a component, we are a cog in a big machine. If we really want to solve agility problems, we actually have to think about the whole system.

Now what you see, and this happens in Scrum training for example, where we see Scrum trainers and they teach the stuff in Scrum and then they teach this other stuff that is not really part of Scrum. Value stream mapping is very systemic, like our little team is here, but what are all the things that have to happen from the time someone has an idea until we actually get our part of the thing to do, until we finally go through all these other people and we finally ship to a customer. That value stream is very interesting and that is systems thinking, in other words – solving systemic problems.

Toyota is famous for having done this. When it first started doing “just in time” manufacturing which was decades ago, it said “We want orders coming in here and then we create these Kanban cards internally. So someone orders a car and now we have to create a chassis and we have the drive train and we have the wheels and all this other stuff. Well, we go all the way to the wheels and now we need to get the tires and, guess where the tires are coming from? A tire manufacturer, outside Toyota.” So they said “Hey, guys. Someone did this order just now and we need the tires.” The tire manufacturer comes back and says “Hey, you know what? We have a quarterly order cycle. You should have ordered that 3 months ago and stuck it in your warehouse so you have it. You cannot just come to us and say you need some tires tomorrow. We are not equipped to do that. Sorry!” And so Toyota could not be Agile. That was a big problem for them and so once they went to this place where they said “Let’s think systemically!” They said “We have to teach the tire manufacturers how to be Agile, otherwise we are not going to be Agile” That is exactly what they did and that is, I think, partly why the Japanese economy, virtually all the manufacturers involved in automobile stuff, are “just in time” manufacturers now and that leaped over into Toyota’s competitors and yet we all kind of think “Japanese cars- high quality cars” That is really great.

It is funny that GM did not learn this, right? They are famous for this Numi plant that they did jointly with Toyota, where Toyota taught them how to do “just in time” manufacturing. It is almost like you can lay this stuff on the door step and you can lead a horse to water, but you cannot make the horse drink. So I think this happened to GM. In 1984 they had this Numi plant that they built with Toyota and they built the highest quality cars that the GM ever built. Did they take the Toyota production system out of that plant and replicate it in all the other plants? They did not. So, 23 years pass and they go bankrupt. Toyota at that time was the most profitable car manufacturer in the world and remains up there all the time. GM went bankrupt in 2007, right? It had 23 years to figure that out and then it hit the bankruptcy wall and died. That is because GM did not embrace these Agile principles. It had a bunch of executives that had built their entire careers out of this non-Agile thinking. They had kids in private schools or whatever and they just wanted to not rock the boat because that could damage their career.

Anyway, now we have five principles. We have the “systemic problem solving” principle. That starts influencing things around you and if you are a Scrum team, the first team that likely gets influenced is things you are dependent on, because you have to do something with that. But then you start having to worry about that “Well, you know, executives are telling me that they have to have this set of features and it has to be delivered at this particular time and when I asked them what is the priority, they say all of these things are priority number 1” That is a classic mess for agilists.

How do I deal with statistics here? And it is all because the executives do not really understand that. If you have too many executives that do that, then you are not going to be Agile. You are going to have classic PMO project managers that are telling you that you have to do it this time and checking if you are on track and if you are not on track, we need to hire more people who are going to confuse things even more and make it even slower.

Now we have these five patterns and now we can asses a company. We can go to a company and say: “Are you measuring your economic progress? OK, that is cool. You have some key performance indicators and they are relevant to your success. Are you adaptively experimenting? Are you experimenting with anything? Many big, behemoth companies are fearful of experimentation. Are you limiting your work in process? Are you trying to punt things out on a frequent basis and then are you adapting to what you see? Do you feel collective responsibility, even the executives, do you feel like you are responsible for the company’s success? And finally, are you solving the systemic problem?”

I guess we would say Toyota is in this camp. They are Agile because they actually went all the way. They said “OK. We are going to solve these systemic problems too” You can do it to individuals too. You can try it yourself. You can go to some individual and say “I am going to try these Agile base patterns with a person” and you say “What are your goals?” and then someone says their goals. “Well, how do you measure whether you are making progress towards those goals?” and if they have an answer, that is a good thing. Probably, they are getting close to Agile. That is good. Then you say “What if you tried to get closer to those goals and how do you measure that?” And they might say “Well, I tried this thing, I tried that thing”. And then finally “How do you limit your work in process? How do you avoid having tons and tons of projects that you are working on to achieve this goal?”

Hopefully, they have someone. If you ask me that, I write and Shane, you write too, so probably you do what I do – one of the things I do is I count the number of words I write per day and I ask “What is interfering with that?” And I start having a picture of that and I know what I can do to structure my day so that I can get the number of words higher. Similarly, I experiment with different things, but I do not as much I should. I am thinking these are challenging questions when you think about them. So how can I experiment with different ways of structuring my life to build better documents that help people? Then finally, how do I limit my work in process?

Well, I do do that. If I have too many writing projects, too many activities, I know it is just a big mess and then I actually, actively cut things out so that I can get stuff done. But many people do not do that and I think that a lot of people who claim they are Agile do not do that. So, I think these are great strategies to figure out who you are talking to and maybe decide who to hire for an executive position because I think that once we get to that place where people are really thinking about that, they are really adapting regularly and they are applying that to themselves or to their family, they are likely to be applying it to your organization too and that is leadership agility and that is why we are calling these Agile leadership patterns for many talks and that sort of thing, but fundamentally they are Agile base patterns. They apply to any Agile thing. If you do them, you are Agile, if you don’t do them, you are not, or you will not be able to retain it. That is what these are all about.

3. Dan, this is really interesting stuff. I know that you have been writing on this and I believe that you are going to be writing on them for InfoQ one of these days.

Yes. I made that promise, didn’t I? So, I have written in detail four patterns. I am a big fan of Christopher Alexander’s pattern style of talking about problems and their solutions. So, each of these patterns is written formally as a pattern. What does that look like? That is: What is the problem? Very succinctly stated and then “What are the forces that constrain a solution?” That can take a while to elaborate, especially because these are pretty broad patterns. Now there is a solution that comes out of that. For example, measuring economic progress is a solution to the problem that we do not know where we are going and we have lagging indicators that are really the measures for success. We have to find some metrics that help us know whether we are on our way. Then there is a bunch of information about what it really looks like when you are measuring economic progress. That pattern is a pretty bug pattern. It is about 4,000 words, I think. One of the things about measuring economic progress if you are in a company is that it ought to be aligned with the goals of the organization. So, what we find is that agility flows top-down, even in the metrics situation. So not only are we dealing with leaders, if we can get our leaders to be Agile, now we get metrics that are going through the organization to be Agile to serve an organizational goal. Now we have more agility and that is why these patterns are there. So four of the patterns are written today and the fifth one is on its way and I promised you that when the fifth one was done, I would write something for InfoQ.

Shane: Dan, thank you very much for talking to us today. It has been really interesting. Enjoy the rest of the conference!

Thank you so much!