Alan Shalloway on Scaling Agile With Lean and Kanban

1. My name is Kevin Brennan. I’m an editor of InfoQ and I’m speaking today with Alan Shalloway. Alan is the CEO of NetObjectives and a co-founder and Board member of Lean Software and Inspection Systems Consortium. Alan, thanks for joining us.

Thanks for having me, I appreciate it.

2. I was told by a few people that all we really have to do is ask you one question and that question is, tell us about why Scrum isn’t enough and why you need Kanban methods.

Well this is funny that you ask me this question because I’ve just actually just resolved this morning not to talk about Scrum anymore. So, let me just talk about what you need to attend to when you’re doing large scale Agile transitions. And what we believe is that when you look at large scale Agile transitions, there are several things you need to look to and that one of them is the culture of an organization; another one is the business side, the management side and the team side and you need to take kind of this holistic view of all of these that organizations or systems. Systems involve the people, involve the methods they’re doing things, involves the structures of doing things and you need to attend to all of these things.

So that means you need to have the technology, in other words, what is software development, what is product development. I saw a tweet the other day about Edwards Deming saying that "It’s not enough to do the best you can. You have to know what to do and do the best you can."

And I think Lean provides a combination of how to deal with the company’s culture, how to create a new method of management, what to look at. Our people say that Lean is really about tools and about efficiency. An aspect of Lean is about efficiency but Lean is really about how you create a culture of excellence and how people do the best job they possibly can and part of that is knowing what to do. So, there’s a lot of talk about flow, minimizing, or not minimizing but managing your work in progress. But a lot of Lean attends to the way people work now in what I call humanity of people.

And I actually think many of the way we do Agile now, there’s this belief that you can just say, "Okay, go for it guys and do it." I don’t believe just saying ‘just do it’ actually does it. You know, you have to actually have to see how could I create an awareness of what needs to happen, how we will solve that problem and the management in doing that; so this holistic approach is really what’s needed; and if you can create that with other methods, then that’s great. I’ve seen that it seems to work really well with Lean doing it.

3. What is it then that you need to do with Lean methods to create that holistic culture?

Well that’s really a good question because, you know, we talk about teams in Agile; a teams is very important thing in Agile and obviously there’s a question of what team is. So, if you’ve got a bunch of developers and testers and QAs working together, that maybe is surely a team, if they’re working together. And sometimes we think of the whole organization as a team.

But in reality a lot of times, our whole organization isn’t a team. People are very tribal in nature, so like you care really more about yourself and your family than you do about me and that doesn’t make you wrong - that’s called being a human being. When you care more about the people you work with immediately in your company, then you care about maybe for people who are far away in your company but that doesn’t make you selfish - that makes you a human being. I do that, everybody does that. So, this is kind of what I mean by the humanity of people, of how they do it.

So if we actually have people with different motivations even if maybe in the big picture academically or abstractly, you want to do the same thing. How do we get a real team across an organization? And that requires a vision, it requires a common not just vision of where we want to go but in agreement in how to get there.

What Lean provides is several things, again it’s a system, it’s not just about tools or methods but one of the methods Lean provides is this agreement that paying attention to time makes sense and it does this in many ways. So one way Lean pays attention to time is the way Agile pays attention to time, delivering value fast is a good thing, both Agile and Lean do this.

Another way Lean pays attention to time is saying that if we have short feedback loops that we will go faster because we will learn faster both the successes and failures.

But I think where Lean is a little stronger is it talks about the recognition that it’s a systems approach. In other words, if you look at a system-by system I mean, what is the way people are working? If they’re working in a good way, then they’ll do good work. Waterfall is kind of a bad way because you had these huge batches and you did not have availability where it’s needed.

So even if you’re good business analyst, you’ll probably weren’t going to get good requirements not because you’re a bad business analyst but because you’re in a bad system. So I think there needs to be some attention to how people working in the environment they’re in and how do you improve that. And Lean is really about creating better systems for people to work in, better environment for people who work in and in my mind, that’s kind of respecting them and supporting them. And it does again through this common knowledge but it also does it through kind of you can say, new management possibility.

In Agile you hear a lot of stuff about, "Oh managers are not helping, seagull management and how they fly in, shit on everybody and fly away." But managers, and I won’t say managers don’t do that; but why do they do that? See, we don’t question that: why do managers do that? We sometimes just think they do it like there’s something wrong with them instead of asking, "Why are they doing it?" And I think the way they’re doing it or why they’re doing it is that they don’t share this vision, they don’t understand the ideas that Lean presents; the flow and how delays and interruptions happen or me telling the team how to do something instead of letting them self organize to figure it out - that’s going to be bad.

So these guidelines are an important part of Lean to create an overall view. In a sense, maybe, to overcome our tribal nature and to create a bigger tribe because now we have a common vision and we have agreement on certain ways of doing this and it also gives us an idea that another aspect of respecting people is acknowledging that people only want to change so much.

So if the organizations isn’t in a good structure, like it’s highly matrix, a lot of companies are highly matrix, or highly silo - how do you get out of that? Do you just jump to the right structure? The right structure, I think, is often cross-functional self organizing teams; but that can be extremely unnerving because you’re now reorganizing.

Think about it this way: you know, I’ve never really said this before but I hear people say, "We were just reorganized." I never heard somebody say, "We were just reorganizing, yay!" It’s usually, "Oh we just reorganized."

So, reorganization is scary to most people. Lean acknowledges this and provides ways of incrementally making your reorganization, that’s what really Kanban is about. Kanban is really about change management system that allows you to start where you are and then make slow improvements. And then it’s doing this not because it’s technologically based but it’s doing this because it actually is acknowledging that people don’t like to change too much. So, if you’re willing to change a lot, Kanban turned up the dial. If you don’t want to change much, create visibility, see what you need to do, create a common understanding and change authority, company, your organization can accommodate.

4. I know one of the key philosophies of Lean is to build a culture of continuous improvement where the team keeps modifying how it works and improving on that. So, how do you do that in software development?

That’s really a question and actually in my mind, one of the cornerstones of difference between Kanban and many of the other Agile methods is that it demands an explicit understanding of what you’re doing, (demand’s maybe too strong) but let me put it this way: the explicit discussion of how you’re doing your work is an essential part of it, that’s a better way of saying it.

So there’s no question that retrospectives and after action reviews on a weekly, every other week, or monthly basis or whatever is a very good thing. And there’s no question that there’s a lot of learning. But the problem is, that’s not continuous and very often if you wait two weeks to actually do your retro, a lot can happen in two weeks and you actually see this kind of retrospection kind of decay after all. I mean, this has been going on forever. Before Scrum was invented, you had companies doing retrospectives, that kind of petered out after all. People don’t really like starting back and reflecting.

One of the things about Lean, if you step back for a second and put some of the things I say together. So I talk about systems. I think I talked about that there are certain things you look at, I mean, there are several, like what I call laws. There are several we know but we don’t actually talk about a lot.

Like here’s one: I’m going suggest that delays of a certain type literally creates additional work to do, for example, if you write a bug and you don’t find out about it for weeks, there’s more work to do than there was if you found out about it right away. Now some of that work might be just remembering what you did, some of that work might be other people had done work and then you go to undo it.

Another way I’ve heard it say, in the Agile community is if you have slow feedback loops, there’s more work to do. Well, that means if you have tighter feedback loops, there is less work to do. So, Lean is giving you the spaces to say, "Let’s keep the feedback loop on our learning kit the way Kanban does it" and you can do this in Scrum.

In fact I just gave a talk at our own conference yesterday about improving Scrum with Kanban and it was still Scrum, but what we said is have explicit policies in the middle of your Scrum. In other words, how do you do the work in Scrum explicitly? So, if you say, "Well, we won’t let something into this sprint unless you have acceptance tests for it and we won’t let it out of the sprint until it’s done and this is what done means. And in the sprint, this is the way we do the work. Now, I can change that this afternoon if I want to. I’m not stuck with it but I’m explicitly telling one team to another, it’s like peer programming works so that it’s very explicit - two people working together talking.

Anyway if you have this expressed in nature, and if you have a board that shows this work, now the board’s not telling you what to do, it’s a reflection of the way you’re working. It’s just a reflection of what you think is the best way. And if on this board, you measure how long you’re stuck in every palm, so you can see the floor work or what’s actually usually better is how much work is waiting to be done. If you have a lot of work waiting to be done, that means it’s going take more time, that means it’s going take more work. Remember time, as it expands takes more work.

So Lean gives you a way. Kanban, in particular, as I’m describing it gives you a way to see on an intermittent basis of how your work is flowing and if it takes more time, it’s going take longer. So on a daily basis, you look at your Kanban board, you see how much work you’ve got in progress. You can say, "Gee, I think we got too much work here, maybe we ought to have or start something else or finish this before we start another one." So like in Scrum, instead of saying, "Well, we might not just pull items at random," which is not a good idea as most people know, say," Oh let me help Paul finish this before I pull another, " kind of like that idea.

Then what you can see is try something, look at where your queues are going and you can make improvement everyday. Now this is not like your daily stand ups, in fact you should have daily stand-ups in Kanban as we all do in Scrum; but in a daily stand-up you’re often just kind of going, "Here’s what I did, here’s what I’m going do" its like a status report and finding impediments which is good.

But in Kanban it shows up a little different. It’s like, "Well we’ve tried something yesterday and here’s what our work progress looks like now. Did it get better or worse? Are we flying faster or not so fast?"

So you have these little mini science projects going on all the time. Some people don’t believe this is possible because we’re complex-adaptive systems, that means there’s no predictability. And I actually don’t agree with that. I mean, I do agree we’re complex -adaptive systems but I don’t believe there’s not predictability. I think there’s been a confusion between what I call micro predictability and macro predictability like micro predictability is if you are in Las Vegas and you spin the wheel, you don’t know what color - red, green or black - there are a couple of green. People forget that. They say red black but what you do know; so, you don’t know micro predictability but what you do know is more money staying at the table on average, that is leaving it.

5. And if they get it down to science, right?

That’s right. So the same thing is with software yes, we’re complex adaptive system but some structures and some methods work better than another and the evidence for this is that if you look at a lot of different Agile teams whether they be Lean, Kanban, Scrum or whatever, they all have patterns of success and they all have patterns of failure.

And the reason they have patterns of success and patterns of failure is because at the macro predictability level, people will show up ad work in certain ways in a better way or worse way than others. Now every situation is a little different so you cannot write a book that say, "Do this in an order." No that’s going work because that’s mirco predictability not you’re seeing a culture that’s not a one-size-fit-all for all cultures. But in your team you can try something and see what are the results you’re getting. If it doesn’t work, stop doing that. If it does work, do more of it.

And that’s an important aspect of things. To me this is not just inspect and adapt because while you are inspecting and adapting, it’s a kind of inspecting and adapting. It’s a kind of inspecting and adapting where you’re saying, "There’s an underlying model I’m inspecting to see how I’m doing against." And the underlying model is saying, "Shortening with timeframe from start to finish is a good thing," Start to finish being the whole thing, not just one coder’s start or end, that would be a local optimization. Global optimizations optimize the whole, means from the front until it’s delivered.

6. On that print can you maybe tell me a little bit about how you use Lean to optimize value stream across the entire business and not just within the development?

That’s a great question because what we find is there are two ways to try get agility at the enterprise level; and one way is to just say, "Well the team is the bottleneck which they often are, so let’s get that better." But what people don’t always acknowledge is, yes, maybe the team is a bottleneck but maybe they’re not the bottleneck because of what they’re doing. Maybe they’re the bottleneck because of something upstream is doing.

So the problem is if the real source of the problem is somewhere else, optimizing the team may get you there but it’s kind of difficult. So our first thing is we actually map the value stream and there are different ways the value stream maps.

We actually have learned that mapping the value stream with a Kanban board -it sounds really weird and it’s something we’ve only been doing for the last year or two but you’re not creating a Kanban board to use, you’e literally just mapping out the work as like a Kanban board because it’s very concrete. People can relate.

Well, some people can be abstract and follow value stream up but everybody can relate to concrete things. So if you actually map out while the work is going on at a high level, on a wall where stick is going through and it’s just like a visualization tool.

7. So like trying to capture a moment in time?

Yes, for this moment, this is what this stream looks like at this moment, then you discuss where we’re blocked; where do we go back, whether we’re there; why do we have too much work; or why do we not have enough work; why do we have not the right resources and you see all this.

We find that companies, their patterns of failures, as we call it are anti-patterns at the company level or enterprise level; some of them are the business team, the business sponsors are not really identifying the best things of value or they’re identifying them but they’re not cutting them down to the core elements, the really important things; or, maybe there’s just too much at play at once; too many endeavors, or maybe what needs to be built you don’t have the right people for; so, you know you need certain skills that aren’t there and you can’t get it all done.

Or maybe the teams don’t know how to do incremental development or maybe the product owners or the portfolio managers are not incrementally giving things to the team, maybe the team has too much technical debt maybe the team doesn’t know how to write code without adding technical debt; maybe the team doesn’t really know the right workflow.

I’m a big believer in acceptance and test-driven development, maybe they don’t really know that they should be getting tests specifications not just test cases- there is a difference. There are like any number of problems although but mostly there are four or five things we find. I look at a hundred companies, I bet 80 of them could be doing one of four things. One of four things would be the problem, the real root cause. So, what you want to do is know that’s what’s going on and then you figure out how do we best adjust that? Now very often it’s a pilot project even if it’s not at a team level, that’s causing the problem. So, by now you’ve picked your pilots different based on the root cause of things. So if your pilot is the team doesn’t know how to write code, that might be one thing and could be a team thing. If the pilot is that the business side doesn’t know how to create the right amount of feature requirements, that they’re asking for two big things, then you focus upstream. And you pick up a pilot that you’d be able to spend more time focusing upstream to get the right requirements to the team but it’s still a pilot but now it’s a pilot for the whole thing.

So where to start or how to start, you got to ask yourself first: what am I trying to do? What’s the root cause? And within that context, you could pick Scrum, Kanban, Flow whatever. But you got to look at the whole thing to decide what to do first and many teams think: "We’re going go Agile so let’s just start with the team project." See, I don’t like that.

I don’t like it even if they say, "We’re going start doing Kanban at a team level" because that’s not a big enough picture of things. Kanban actually move you up the chain quickly but you always have to look at the whole. Even if you don’t have control of the whole. I was just thinking as I stop then I said, but I know what other people always tell me, "Oh but Alan we don’t have any control over what the business people are doing." Okay, great but it’s good to do a little bit of analysis, start where you can, create visibility so that they can see the impact of what you’re doing.

8. Now that reminds me of something I actually read about on your blog. You talked about how certain types of changes produce essentially big change up front and can you talk about why that’s a bad idea and what you would be doing instead?

Actually big change up front isn’t always a bad idea but my point in the blog was you have to ask yourself what’s the right amount of change. And I was doing a play on the big design up front. So BDUF isn’t good, we all acknowledge BDUF isn’t good then why is BCUF good and when I say BCUF I mean big change up front.

If you’ve got a matrixed organization and create cross-functional teams, basically put the Scrum structure in and the Scrum structures are great, there’s no question about it. If you get people to work in cross-functional teams where they’re not being interfered with by management, they get to pick how much work they do in four weeks and they work together. I mean, if you do that, you should get at least five times improvement from a matrix organization because we found that if you have a matrix organization and just create collocated teams that are cross- functional, and they get to do what they want even if they do waterfall they’re going to be three times better than siloed teams. So if you’re not getting five times improvement going from matrix to Scrum, there’s more opportunity there.

So the question is can you make that jump? Now in some cases you can and in some cases you can’t and there are two reasons why in some cases you can’t make the jump. One case is fear. You know, if you look at the physiology, now this is a human nature of things. People can attend to so much change and then they get nervous.

Now everybody quotes Gerry Weinberg, I think Gerry is one of the nicest, sweetest and smartest guys - great combination and he talks about Virginia Satir all the time. Well, Virginia Satir she’s a very noted psychologist; and she talks about people being in comfort zones and they will go to their comfort zones. If you jump them out of their comfort zone they will react with fear and they’ll resist change. So if you jump too far and then you come back, the next time you want to jump, you’ll jump less far and be more resistant. So jumping too far goes against human psychology if you do it too much.

I’m not saying, "Don’t make the jump," because I’m not against big change up front. I’ll give you an example. One time I went to work with a company. We do this thing we call Lean and Agile transition workshop, we spent four days, we work with leaders of the organization-this was a seventy-person group in a big company, this was all product and we have about 25 of their team leads, tech guys, managers all together for four days; and basically we talked about Lean and Agile and we came up for them or they came up with me guiding them a whole new way of doing their development process. They went from basically non-Agile to a Agile but neither Kanban nor Scrum, it was custom based one-flow and we had four more days of coaching.

And so, 70 people like that, that’s all we ever did for them, four days training, four days coaching. And the results were remarkable. Now, why did that happen? Because I look at this. Why did that happen? Was it just a good day for me or was it a good team? Well, it might have been along that day but I think what it really was is you had a team that got to understand their problem by looking at Lean, their problem turned out to be integration; their problem wasn’t that they had big stuff coming in; their problem wasn’t that they didn’t have a code; their problem was they had 50 developers and 4 -7 at different times test platforms, in other words, they write the code but they couldn’t test it. So, you have 50 people sharing 4 -7 test platforms - pretty complex stuff as well.

And once we saw what the problem was by looking at flow, by mapping it out, then they were all able to kind of create a large cross-functional team. Now it’s funny they still have different silo teams - there were seven components, I know, there’s a military aircraft; and you still have seven teams but they were now able to dynamically reorganize when something came in that needed instant integration and the work done.

Now they were able to do big change up front but only because they had a common vision, they have an understanding of why they were doing what they’re doing and they could see the result and to be comfortable about the risks but if you don’t have all those things in place, it’s a lot better, (I’m not saying don’t change) I’m saying take any indication on how much you can change and then don’t change more than that.

So if you can organize the cross-functional teams, do it. I’ve said there are two reasons why cross-functional teams are tough sometimes: one is the degree of change. The other one is actually some people are necessary to work across multiple teams. This idea of cross functional team is just wonderful. People talk about skills. Sometimes you have skill sets that are hard to replicate or to cross train.

I worked with one company where the starting point for some function was PhD and then 10 years of experience because that was stress test analysis. If you have a hundred people that only need one guy like this to share across them, you’re not going to cross-train and then you’re not going to hire more because that’s too expensive. So, how do you manage that?

Or, a skill that’s sometimes more common is domain knowledge. If you need somebody who’s got 20 years of domain knowledge, you don’t hire him and give him 20 years of domain knowledge in two weeks.

And the third one is legacy code or knowledge of it. Some companies, the most valuable person is the guy who was there ten years ago when the code was written and he knows why and nobody else does. And you don’t get that kind of experience.

So it may be impossible to get the structure that you need and therefore, big or small change up front is not good if you have this kind of impediments and overcoming them is neither financially or emotionally a good idea.

9. Alan, we’ve got time for about one more question: if it’s one thing you could tell people that they need to be taken cared of, what would it be?

Well I would suggest that people should look at what they’re really trying to accomplish and I know that sounds like a silly question, of course, you know what we’re trying to accomplish. But you know the thing is, are you trying to accomplish team agility which is a worthwhile thing and it means getting your team to be agile. Or are you trying to accomplish product agility which is a little broader, maybe department agility or enterprise agility and what do you need to attend to to get that done.

So there are many frameworks out there, methods out there, processes out there and I really do not believe they all work equally well at all those levels. I do believe one of the interesting things about Lean and Kanban is it can work at all those levels. There are things you could do that can make it not.

But I think that to me, if you say, "Well, we really want to just stick with the team", then using a team centric approach is a good idea. But if you’re trying to go beyond that and it’s not improving the team, that’s not necessarily improving the entire organization. In fact in this case where I mentioned, this group of 70, like in four days of training and four days of coaching, went into magic transition. The reason that happens is one of the teams acknowledged that they had to slow down; and for them be less efficient, but with the help of the other teams in the whole organization to go faster. So that team wasn’t trying to get better, that team was trying to help the other teams get better. So, you have to look at what’s the scope you’re trying to go.

So if you’re trying to get enterprise agility and you only have team control, but you want enterprise agility, then ask yourself: how can we improve what we’re doing in such a way that it will improve the bigger picture and that’s often ignored. People are always putting walls around them and they don’t allow interference and this lack of, they may stop the interference but they also stop influence. They can no longer influence the other organization. So I would say, you want to ask yourself how can we accomplish what we want while we influence that part of the organization we don’t have control over.

Personally I’m pretty optimistic that that’s possible and I say that because I’ve seen it. I’ve seen teams change organizations when they enabled the organization to see what they are doing and they looked at the bigger picture and the context.