Mary, Tom can you please introduce yourselves and tell us what you're working on?
Mary: My name is Mary Poppendieck and I'm working on the concept of Lean Software Development. I was a programmer for many years and then I went into management, into product development and got out of software development for a while, and then I came back in again after I left the company that I was working with. I was involved as a Project Manager and that's the first time I ever ran into this idea of "waterfall". I said to myself: "How can this possibly work?" Then I discovered it actually didn't work, so I decided to figure out how to apply the concept of Lean, which I'd worked with when I was in manufacturing, to software development.
Tom: I'm a physicist. I've been a professor, a high-school teacher; I've worked in industry - working on navigation systems for commercial airplanes. I've been in small and large, and I've spent the last part of my careeer as a software consultant and encountered many of these ideas as they were being introduced in the last decade. Currently we're working together to spread the ideas of Lean, as they establish a context for Agile software development
Where does Lean come from?
Mary: Lean comes from the Toyota Production System which was invented at Toyota for automotive manufacturing in the late 1940s, 1950s, 1960s. We didn't actually discover how it was working in the US until perhaps the early 1980s, when the idea of "Just-In-Time" manufacturing started competing against other US products, so we started seeing Toyota and other Japanese cars taking market share away. In my industry, which was 3M, we were making video cassettes and all of a sudden we found that Japanese competitors were selling video cassettes for a third of what we could selling for, and less than we could make them for. We were trying to figure out what caused that. It turns out that this concept of Just-In-Time manufacturing was strongly behind what was going on, and later the concept became known as Lean manufacturing.
What is Lean software development?
Mary: We'll talk about Lean in general. The Lean history starts over here in manufacturing. Here the first thing Lean was known as was "the Toyota Production System" (it was the way Toyota learned how to manufacture cars). That became known as Just-In-Time and that how it was known when it came to the US and Europe. Then, in 1990 a book was published "The Machine that Changed the World" - the Story of Lean Production. That's where the word Lean production came from. They're all basically the same thing. They're a way of thinking about manufacturing that allows you to do rapid manufacturing with very low inventory and high variability. Then, if we come down this path there's something in logistics (warehousing, moving materials between companies) called supply chain management (SCM). SCM is the way that you use Lean in the logistics area. And if we come over here there's a whole other area which I'll call Product Development, which is very different than manufacturing and logistics. We now apply Lean thinking principles (not the practices, but the principles) from manufacturing and logistics into Product Development. When you apply Lean into Product Development you get a different way of looking at it. And I believe that software development is a subset of, is like, it's part of Product Development. When you apply Lean to software development you take the general principles of Toyota production system or Lean and you apply them into the Product Development environment but you don't use the exact practices that you would use in manufacturing. You have to go back to the first principles of what you're trying to accomplish and move those into software development.
What are the main principles behind Lean?
Mary: The main principles behind Lean were articulated by Taiichi Ohno, the person at Toyota who invented the Toyota Production System. The first principle would be the idea of Flow (or Low Inventory, or Just-in-Time). The second one is what I would have to call "expose problems" or "no workarounds". The idea is that you have flow and you have low inventory: it's like you have a boat on the water and your boat is sailing above these big rocks, which are problems. This is your inventory level here. If you lower your inventory level, at some point you're going to run into a rock rock and your boat is going to come down and bump into this rock. If you don't get rid of these rocks when you lower your inventory you're going to bump into rocks, and crash and burn. The first thing you're doing is lower your inventory to expose those problems so that you can get rid of them. If you don't expose problems and stop and fix your problems, then lowering your inventory is just going to crash your boat. You have to do two things: you have flow but you also have no tolerance for abnormality. You just don't allow defects into your system; you don't allow things to go wrong. When something wrong happens you stop, you figure out what is causing it, you fix it rather than continuing on and just ignoring it or working around it. These are the two basic principles.
What does "inventory" mean in software development?
Mary: In software development inventory is anything that you've started and you haven't gotten done. It's "partially done" work. In manufacturing if you start making something and it is in-process, it's not sold, it is inventory. In development it's the same thing. If you started developing something and it's not done, it is inventory. What you're trying to do Lean software development is the least amount of "partially done" work as possible. You want to go from understanding what you're supposed to do to having it done and deployed and in somebody's hands as rapidly as possible.
Tom: "Done" means coded, tested, documented, integrated, "Done", so there's no more work to do. It is one of the main reasons why Lean approach gives you a more reliable delivery, a more trustworthy delivery. So that instead of finishing activities like requirements, which doesn't tell you much about how far you are along overall, each piece of work that you do is completely done.
What are some of the other principles of Lean?
Mary: The way I apply Lean into software development, I have seven principles that are the foundation of the way to look at it Lean in software development. The first principle is to eliminate waste. The second is to amplify learning. The third is to delay commitment. The forth is to deliver fast. The fifth is to build integrity into the product. You can't test it in later; you have to build it with integrity in the first place. The sixth is to engage the intelligence of the people who are doing the work. The last is to optimize the whole system, not just part of it.
What is "waste" in software development?
Mary: Waste is anything that the customers do not value. If you're doing something and in the end the customers don't think it is important for them, then it is waste. There are lots of ways to look at waste. One is: partially done work is waste. Just like in manufacturing, inventory is waste; in software development partially done work is waste. Also, extra features you don't need right now is waste; stuff that causes delays is waste; things that get in the way of a rapid flow of product are waste. Because customers, when they have a problem they want the problem solved now, and stuff that delays that is waste.
What does this look like practice?
Mary: We start by asking people to draw a Value Stream Map. You start with a customer problem-need request, and you go to where that request is filled. So, you put on "customer glasses", and now I want to watch what happens to that problem until it is back and the customer problem is solved. You draw a map or a timeline of everything that happens from the time the customer request comes in the organization until the customer has their problem solved. You lay out the activities there and how much of the time are you really adding customer value and how much of the time is just sitting there contending with other work that has to happen.
So, one of the things you do to eliminate waste is to say "From customer request to customer delivery, what is my cycle time? How fast do I do that, reliably and repeatably (not: every so often I can do it this fast). What are the steps and how much of that time am I really adding value?" You do that and you find out that, for instance, you develop these requirements... Oh, first of all, it takes forever to approve it, because some how or other you don't think approving is important, and then you develop this set of requirements, and after a while you discover that you have to change them. So, you go back, you loop back, you've done a lot of work that's isn't even needed because it gets changed, and then you actually code it.
Well, when you have requirements churn, meaning you've done some requirements work and then it needs to be changed, that means you've written your requirements too soon. If you don't wait until it's time to write those requirements (that's what I mean by "delayed commitment") that's when you're going to have a lot of changes in requirements. On the other hand say you get to testing and you find errors - oh my goodness! Now you have to go back and code-and-fix, and code-and-fix. If you have that cycle in your value stream you're testing too late. The idea is to move the detailed requirements closer to the coding, the testing closer to the coding, and do it in smaller chunks.
What it looks like in practice is an iterative cycle that's 2 to 4 weeks long in which, just before the cycle begins, I take the top priority things that I want and I blow them out into detailed requirements. I write the test first, because the test really helps me understand the requirements, and then I code so that I pass those tests. I make sure that I'm documented, and done and ready to deploy. That means I've integrated it into the code base, on a constant basis; if something goes wrong I stop and I fix it, I don't build up a bunch of defects.
I actually don't have a defect list, it's much better to fix the defects, not that you don't have a couple of those that you decide to ignore, but generally speaking you don't even have to have a defect list: you need to have a practice that says"When I find something that's wrong I fix it." It's either wrong or it's right and I don't have to say "I'm going to have things that I can fix later". So you write the test, you write the code to pass the test, you integrate every single hour or two into the code base; when something goes wrong you stop. Now this requires automated testing. It also requires a lot of discipline.
When it's done I have already tested and integrated it. Yes, I do a validation stage but I shouldn't be finding errors in the validation state. I should generally pass my validation stage and then I can deploy it. Probably automatically, and with something that makes the customers feel safe. This is very much like maintenance programming where maintenance gets the error, and if it's a serious error in 24 hours they have that thing fixed and patched and back in production. This is what I mean by a rapid cycle.
What does it look like when you have a large team, let's say 30 developers?
Mary: When you have 30 developers you have three teams and not one, or maybe even four teams. Or maybe you've tried to create a divisible architecture so that your architecture allows individual pieces of smaller teams to work on that. Then that's coordinated by a few people from those teams.
Tom: But even if you've broken it down into three teams they all work on the same code based so they'll continue integrating and synchronising with each other. If it's a distributed environment you still have one code based one repository. People that live near the repository get faster builds, the ones that are remote get slower builds.
What impact do delays that are outside the development group have and what can you do about that?
Mary: I'd like to stop and say that the development organisation should not be an independent thing. The development organisation is trying to make something that's bigger than the development organisation. If you're just developing software you're actually just developing something that's probably pretty useless. If the software is not embedded in a business process or in a product it is really not very useful. The first thing to think about is: the development team is part of a bigger team, one that is trying to change a business process or that's trying to put a product on the market. You need to look at the development team as a piece of that whole team that's trying to deliver value to the customer. There has to be somebody that is looking over the entire process and making sure that all the different organizations are working together to create that fast value stream. But I admit that every time we find value streams with big pauses and big delays in it, guess what: it's almost always true that you're crossing organizational boundaries at that point. Nobody seems to own moving the process forward across the organizational boundaries. So, you need to have an environment in which those organizational boundaries have somebody owning it, making sure there's coordination across the whole thing you're trying to do; the whole product or the whole business process, not just the development part. Otherwise you can develop forever and not actually have something that's an overall business success.
Tom: The bottom line here is that there's no such thing as technical success; there's only success - and success means that the business benefit, that ROI that justified this, is realized success - this means that this product sells on the market. Technical success is irrelevant. It might make individual people feel good, but if it doesn't succeed in the market, in delivering ROI, it's a failure.
Mary: So one of the things I'm looking for when I say "optimize the whole", is a team that looks at the whole picture and not just the development portion of it; and a leader that leads the team across the entire value stream, not just the piece of it that's "development."
You mentioned one of the principles as "engage intelligence of the workers". How did you do that? How does that look like?
Mary: It means first of all, understanding that the people that are doing the work are the people most capable of figuring it out how to do it best. So if you have a process, it is not designed by some organisation that you might call " Process Police," that figures out how things should be done. You have the people that are doing the work figure out what is the best process to do the work. You don't pretend that management is the people with most intelligence to figure out how things should be happen.
You create what I would call a visual work space so that when somebody shows up for work in the morning he can look around, see what needs to be done, figure it out and make it happen. Some of the mechanisms to do that is, if you have a team working together and at the beginning of the iteration they decide together what they can do over that iteration, then they take cards and post it on the walls and when they walk in they say "Ok I'm going to do this story," etc. They can see what is done, they is not done, they meet daily to figure out how they are going to solve the problem and at the end of the two week iteration they have it done. Nobody is tracking each task and telling the team how to do every individual thing; instead the team is being chartered to figure out how to make things happen themselves and how to make things best. When I was a manger we were trained that the most important thing is to make sure the people who are doing the work are the ones that know it best; if you think that you can make decisions for them: guess what, you'll make more mistakes than they will. You need to structure the work so that it can be done well by the people doing the work and so that they can figure out how to do the best thing. That's what I mean by engage the intelligence of the workers. The team should be doing the project control, they should be deciding what tasks should be done next; you shouldn't be having somebody telling them what to do.
Tom: The opposite of that is conformance to somebody else's plan, to somebody else's process. It isn't that there isn't a process, but the process is owned and evolved and continuously improved by the people using the process, rather than having a process imposed.
This sounds a lot like Scrum. What's the difference?
Mary: Scrum is a fine example of a Lean environment. Scrum is a set of practices; this is how you do things. Lean would be the principles behind those practices. Lean is the general principles that encourage you to use something like Scrum. Lean would basically say: "You can decide in any given environment how that principle should be applied." So what it means to have delayed commitment? In one organization it might be different than in another. With delayed commitment, for instance: don't make decisions until they have to be made. In some Scrums you have a fixed two week window, in some organizations, maybe you really can delay commitment until a week before. Maybe if you're doing embedded software you have to synchronize how you do things with the hardware team. And so exactly how you do things will depend upon the context in the environment, but the underlying principles apply, and if you apply the principles you will get that Scrum it's definitely a Lean process, for example. But there are other processes that also could qualify. For example a lot of "open source" development could be conceived as quite Lean, but doesn't actually conform to a lot of the other principles. There are many example of how Lean can be applied and Scrum is a really good one.
Tom: The difference between Scrum or XP and Lean is that Lean looks over the whole value chain and allows you to understand some of the impact that organisational boundaries have on your efforts, when there are boundaries between testing organizations and deployment organizations, requirement organizations and so forth, and things get handed off from one group to another. There are delays, there's information lost, there's lack of feedback, and all these things are exposed when you start with the value stream. They're not really visible when you measure only a little piece of the process, which software development methodologies alone tend to do.
What popular ideas and processes are directly in conflict with Lean and what is your position on those?
Mary: One of the popular processes that is not exactly in conflict but comes from a different industrial paradigm would be CMM. CMM has this concept of breaking things down into little tiny pieces and measuring every single little tiny piece. That comes from the good old scientific management days, it's a Taylorist concept where you break things down into little pieces and you optimize every piece rather than the whole. Instead of having hundreds of detail measurements, Lean would focus on a few key high-level measurements, like cycle time. For instance, from the time I get a request to the time I deliver it, if I can rapidly and repeatedly do it in a two week window or a known time frame, I have to have all of the disciplines that CMM wants me to have in place, in order to do that. But I wouldn't necessarily be measuring every single one; I would know that my overall measurement shows me that I've got my process in control.
In manufacturing a mature organization is one that has a fast repeatable cycle time, and the definition of maturity is rapid repeatable cycle time; that covers all the other disciplines that have to be in place. I define maturity as repeatable, short cycle time. Is not that a Lean organization is not capable of receiving CMM certification because most certainly probably would be and because you can't go fast repeatedly without having the disciplines in place. Instead of dividing things in detailed things and making sure that every individual piece is done, you look at one overall measurement and that drives all of the other things to be in place. I think CMM comes from mass-production paradigm of dividing things into little pieces, whereas Lean comes from the paradigm of having a more overall look at the whole measurement instead of the detailed pieces.
Tom: Cost accounting is aligned with CMM: measure the cost of each little part and add it up. Throughput accounting measures "how much money did you make from your activity?" That's Lean
What about Rational Unified Process? How does that fit in?
Tom: The rational unified process, like CMM, has a very significant amount of wisdom, that is distilled, systematized and organized. The value of it can be very large as an aid to an organization in learning techniques, as checklists. But the way that is adopted is unfortunately very often a bureaucratic and compliance-driven, rather than a "Lets get the parts of the wisdom , that are applicable to our current context." It can be very valuable but it is packaged in a way that is difficult to find the parts you need, unless you are an expert. And if you are an expert, you don't need it anymore.
Mary: There's a concept that RUP people say: that you configure up and you can only use the pieces of it that are applicable to your area. In practice that does not happen. People don't do parts of RUP, they do every part of RUP and although that is not necessarily the way it was meant to be, it is definitely the way it is oftentimes implemented. So, you can't downscope and take the good parts. It seems you have to take everything whether it is truly applicable or not. Once you do that you're adding a whole lot of waste into your process, stuff that is not really necessary. There's no focus on "what of this can we not do," because it's waste. The same is true with CMM. "What of these pieces do we not need, because we are already there in those places?" or "We don't need that piece for this context." It's sort of like you have to do every little bit and let's look at the overall thing to see which pieces are going to give us the best result. Which of these pieces is just plain waste, just stuff that doesn't have value from a customer's point of view?
Tom: Unified Process is also based on a set of principles which are not totally dissimilar to the Lean principles. But the principles are suppressed and ignored and the focus is on the practices, and the artefacts, and the documents and the processes. If the adoption of the RUP would focus on the RUP principles, as we advocate focusing on the Lean Principles, the likihood of success, leveraging the intelligence of the people rather than encouraging their conformance, is far more likely to be successful. It is not RUP vs. Lean vs. Scrum vs. XP, it is the mindset of the individuals and of the organization that matters; which is why we focus on the principles and how it applies in individual contexts rather than on individual practices or artifacts or anything else.
What is "delayed commitments", one of the principles you mentioned?
Mary: The idea of delayed commitments is not to decide until you have the most information that you possibly can have. Then you make decisions based on that. It's the idea that (not that you procrastinate and never make decisions) but that you schedule decisions for when you have the most information. For example, pilots are trained that when they have to make a taught choice, they should decide when they have to decide, and not to decide until then because that's when they have the most information. The military paradigm: when you're threatened decide when you need to respond and don't respond ahead of that, wait until that time because then you have the most information. But don't wait any longer than that. So, it is the idea of scheduling decisions until the last moment when they need to be made, and not making them before that, because then you have the most information.
For example, let's take user interface. When do you really need to design a user interface? Oftentimes it drives the whole design, but in fact you don't really need it until you're about to do your first alpha test. Before that you can be designing the business layer and you can actually put testing in below the user interface and you can be designing all of the other business logic; you can get that done with any kind of interface and in fact you ca drive testing with a automated interface, and then just before you go to alpha testing you decide what you want for your user interface. Then you take it off and at that point in time you figure it out. But up until that point in time you don't need that. And there are many pieces of decision in software development, where there's this idea that we have to design the whole thing before we get started. But the fundamental Lean principle in product development is that we should not make any design decisions until we absolutely have to. We really do not want to have decided anything until we need to; and so at the point it time of the decision we should still have 3 options. Still have a couple of middleware options, or still have not decided how we're going to do the user interface. You wait until you know the most possible information before you make decisions. So, the idea of having a complete detailed spec at the beginning is the exact opposite of the Lean commitment.
Tom: At the beginning is the time of maximum ignorance about what you really need. It is hardly the time to make your key commitments. Most of the cost is incurred when you make the first commitments. If you can delay that, you can do a much better job.
Mary: You want irreversible decisions, decisions which you can't change, to be made as late as you possibly can. Otherwise, if you make them early you're going to lock in the decision, because you're going to build stuff on it, and you didn't need to make it yet. And if you wait, you leave options open so that you can chose later exactly what you need to do. That's just basically a good design heuristic.
Another principle you mentioned is "deliver fast". Shouldn't we be slow and more careful?
Mary: Well, I don't believe, and there's a lot of evidence to show, that being careful, having high quality, is most certainly is not related to how slow or fast you are. If you take a look at companies that are really good, they also tend to be really fast. Take a look at how fast Dell can deliver a computer. You can't do that if you're sloppy. Take a look at how fast Fedex can deliver a package. Only companies that really have their act together and are disciplined technically can be fast.
In Lean the whole idea is that you can have variability with speed and quality. You can't go fast without having everything together. PatientKeeper, which is Jeff Sutherland's company, delivers 45 cycles to its competitor's one cycle. Now, they can't deliver, they die if they have defect-ridden code. They have to have good stuff. If they don't have a slick, workable installation process they couldn't do it. They can take one code base, they don't have a lot of branches, ant therefore they have mandatory updates for all kinds of their customers, who are hospitals. If they didn't have software that their hospital trusted, and allowed them to have continuous upgrades, they could not get away with it. So if you're going to be fast and rapidly respond, you'll have to be very trustworthy. Just like the maintenance department. Let's say you have a critical error and your system crashes. Your maintenance programmers go in there, find out the problem, they figure out the fix, they test it and they put it in the patch and you let them do it all the time. That's because they're fast and they're also trustworthy. If you think about software, why is other software any different than fixing up a critical error? You should have all of the same processes in place that a good maintenance department has so that when something goes wrong you can detect it, you can fix it, you can put it in the patch and you can do it where the entire organization has confidence in your capability to do that. Real true speed is actually something that has to come with extremely high quality. So, I really contest the concept that you've got to go slow and be excruciatingly careful. What gives you confidence is excellent testing procedures, is automated testing, is stuff that's disjoint from each other so that as you add features you don't add complexity. It's simple code and very disciplined procedures but not go slow. In fact is have your automation there, so that you can go fast. Then you can have higher quality than "let's be slow and careful".
This sounds nice but is it realistic in large organizations with a lot of boundaries between departments?
Mary: We get people saying a lot: "I can't do that in my company". It is very true that the organizational boundaries are where this falls apart so we'll give a class in Lean and people come back and say "I can do it until I run into testing, but the testing department does not want to have anything to do with it" or the people in requirements, or our customers don't want to think about things differently. As you get up to boundaries you really have some problems.
But organizations that are in a deeply competitive environment tend to figure this out pretty fast. Because, if you get a single competitor out there that can do this, they can probably knock your socks off real fast. So, for instance, PatientKeeper puts out 45 cycles to its competitors' one, and all of a sudden your competitor can flood the market with really good, high quality, constant improvements and you can't - you're in trouble. What we find is that the organizations that figure out how to do this are the ones that have threatening competition and they have to be able to do something that is really new, novel, different and highly competitive. If you don't have a lot of competition and you can keep going the way that you're going and you don't have to think about how your organization is structured, you might actually not be able to do Lean in your organization. But when you got into competitive companies where you have got to be creative and you have to figure out how to hit the market better than your competitors, this becomes a very nice competitive advantage. And, the companies that we see using this are the companies actually forced to think really hard about how they can have some leverage their competitors can't have.
Tom: We're seeing both very large and very small companies using these ideas. This kind of thinking is largely responsible for the success of Dell, Wal-Mart and other small companies like that.
What do you do when you can't get Lean into your organization? What's one approach?
Mary: If you run up against an organizational boundary that you can't do, then Lean is not coming into that company at a high-enough level. Very often it is said that you really have to have Lean as a corporate philosophy, that's in the blood of the whole management structure, otherwise it tends to have problems. So, you really have to think about this as being a fairly high-level corporate philosophy or culture, before it can actually work across the various organizational boundaries you might run into. Not all companies want to switch their culture that way.
On the other hand, there are a lot of Lean initiatives happening in our world these days, in other areas, not software. You'll find it in operations, in retail operations, in warehouses, in manufacturing and in business processes - Lean has become an initiative that various companies have. One day that finds its way into the software development environment, and when it does at least you have a management structure that is beginning to understand what Lean means. And if you translate it correctly into software development, you probably can get some serious management support for Lean initiatives, also in software.
How do you do Lean under contract?
Mary: Ah, well that's really a problem. All of the Agile techniques run into trouble when the organizational boundary that you're crossing is into another company. Because, when two companies start working together they tend to need to have a contract between them. And the contract usually has to spell out exactly what were doing, and exactly how much it is going to cost; most contracts then create a game between the companies, where you have predefinition of everything, and you have to check it all off later. If you have a contract like that it is very difficult to have a Lean environment, or any kind of Agile environment.
But there are now some contracts that are being formulated that recognize that Agile gives you better results. For example, Norway has a PS2000 contract, it was written by the Norwegian computer society, and it is a standard contract for a large public service, public IT contracts, and it is specifically written to encourage Lean, or Agile, software development. It defines how the parties are going to interact together with the assumption that the parties are going to figure out on an outgoing basis what it is that they really want to do. And it creates a contracting environment that allows people to "inspect and adapt" as time goes on, and have feedback in the entire development process. But most contracts don't do that, and a contract that doesn't allow feedback, change, that requires everything to be predefined is going to make it very difficult to do any kind of Agile process at all.
Any last words on Lean for the audience?
Tom: May the Lean be with you.
Check out Tom and Mary's latest
Lean book.
3 comments
Watch Thread Reply