New-age Transactional Systems - Not Your Grandpa's OLTP
John Hugg discusses high volume transaction processing applications with high and low frequency profiles, and how VoltDB can be used for that purpose.
News
The content has been bookmarked!
There was an error bookmarking this content! Please retry.
Article: Lean and Agile, Marriage Made in Heaven or Oxymoron?
Posted by Amr Elssamadisy on Feb 13, 2009
Dave West takes a look at the world views of the Agile and Lean communities and finds them in conflict.
Lean views software development as a process for moving from conception to product. It wants to optimize that process, albeit in a radically different way and with radically different values than traditional (e.g. Taylorism) attempts at optimization.
Agile views software development as a process for building a consensual theory of the world: with an artifact being a byproduct - an expression - of that theory.
Because the fundamental worldviews of the two sides are dramatically different, it is inevitable that there will be conflicts. These conflicts will usually manifest themselves at the level of tools and practices.
If true, then many of us in the community blending Lean and Agile and unaware of the inherent clash in ideals could be making some big mistakes. As an example of a manifestation of this conflict Dave takes the backlog:
Lean looks at the product backlog and sees 'inventory' and 'waste.' Mary Poppendieck is on record suggesting that the product backlog should be eliminated or, at minimum pared to a size more evenly matched to the collective velocity of the teams'.
Agile sees the product backlog as a snapshot view of an emerging theory. Even if that snapshot view is physically manifest as a wall full of story cards, it is not an inventory! The cards on the wall serve as a form of external memory, with each card evoking (recalling to mind) detailed conversations and understandings of how things work.
This article is a must-read if you've been dabbling in both the Lean and Agile worlds.
- This article is part of a featured topic series on Agile Techniques and also Agile
- See more Agile Techniques content at: http://www.infoq.com/agile_techniques
- See more Agile content at: http://www.infoq.com/agile
- Other recent content items in this topic
RelatedVendorContent
Case Study: IBM's Agile Transformation
Agility at scale, become as agile as you can be
A Guide to Branching and Merging Patterns
Related Sponsor
Maximize your business-responsiveness with Mingle. Provide your global development team a shared space that adapts to the way they work.
Yep! Backlogs are essential.
by
Amr Elssamadisy
Posted
Re: Yep! Backlogs are essential.
by
Lisa Crispin
Posted
Lean - Misunderstood
by
Paul Beckford
Posted
Law of Software Process
by
Michael Hunger
Posted
Tangible, viewable, and negotiable
by
Mouneer Rabie
Posted
Lean and Agile: should cousins marry?
by
Steve Freeman
Posted
Re: Lean and Agile: should cousins marry?
by
Paul Beckford
Posted
Re: Lean and Agile: should cousins marry?
by
Paul Beckford
Posted
Lean is more than production
by
Russell Healy
Posted
Re: Lean is more than production
by
Paul Beckford
Posted
more pointless & damaging misrepresentation
by
rob bowley
Posted
Re: more pointless & damaging misrepresentation
by
Andrew Walker
Posted
Re: more pointless & damaging misrepresentation
by
Paul Beckford
Posted
Re: more pointless & damaging misrepresentation
by
Jeff Santini
Posted
Misleading
by
Sebastian Kübeck
Posted
Re: Misleading
by
Paul Oldfield
Posted
Things turning a little too negative?
by
Amr Elssamadisy
Posted
Re: Things turning a little too negative?
by
dave west
Posted
Re: Things turning a little too negative?
by
Jeff Santini
Posted
Isn't a huge backlog a truly waste?
by
Guilherme Lopes
Posted
Re: Isn't a huge backlog a truly waste?
by
Amr Elssamadisy
Posted
Re: Isn't a huge backlog a truly waste?
by
Guilherme Lopes
Posted
Discuss programming as theory building
by
Jason Catena
Posted
Just not practical
by
Jack Milunsky
Posted
Valentine day anyone?
by
Olivier Gourment
Posted
Lean does not wear “production glasses" - people do.
by
Jay Packlick
Posted
Re: Lean does not wear “production glasses
by
Arun Batchu
Posted
Programming as Theory Building
by
Mary Poppendieck
Posted
Backlogs don't necessarily serve effectively as a theory of the product
by
Jason Yip
Posted
Re: Backlogs don't necessarily serve effectively as a theory of the product
by
dave west
Posted
Re: Backlogs don't necessarily serve effectively as a theory of the product
by
dave west
Posted
Re: Backlogs don't necessarily serve effectively as a theory of the product
by
Richard Karpinski
Posted
-
Agile sees the product backlog as a snapshot view of an emerging theory. Even if that snapshot view is physically manifest as a wall full of story cards, it is not an inventory! The cards on the wall serve as a form of external memory, with each card evoking (recalling to mind) detailed conversations and understandings of how things work.
I have never quite looked at a backlog in this manner before - but it rings TRUE. I don't know about others, but I've been really uneasy ever since it became cool to diss backlogs. -
Lean has become fashionable amongst the Agile crowd that like to think of themselves as being on the bleeding edge. My observation is that many of the Lean practices are misunderstood and applied out of context.
Take waste for example. What is a wasteful activity on a production floor is very different then what is wasteful in a creative new product development environment. I have heard people talk about retrospectives as waste. Taiichi Ohno surely wouldn't agree.
The bottom line is that Toyota came up with the Toyota Production System, by continuously asking tough questions, and being determined and disciplined in their efforts to streamline and improve motor vehicle manufacture. Taiichi Ohno's number one tool was "Five Whys", repeatedly questioning practices to get to the root cause of problems and to gain new insights.
Applying the lessons they learned out of context is not to understand the TPS philosophy at all. The TPS took Toyota over three decades to perfect and they are still improving and challenging assumptions. Good old fashioned hard work and graft may be unfashionable, but Lean is no silver bullet and its not a free lunch either. We still have a long way to go in discovering what lean principles and practices if any make sense in the context of software development. It is too early to say because we haven't done the work yet!
Paul. -
Regarding the nature of Software Development - The approach that Phillip G. Armour takes in Laws of Software Process - i.e. that Software is the fifth kind of storing knowledge (and make it executable by the way) is also a quite interesting model.
I think there is no one right or wrong you always have to see it in the context. So if a principle or practice helps to improve your efficiency it is no waste. And backlogs do as the knowledge they assemble would have to extracted again and again from the customer.
Michael -
Although we have not been with agility for so long, but every team member can attest to the value of the viewable, tangible and negotiable product backlog. By no means, the backlog can be considered waste; after all and so far, it has been “working” for us.
Totally agree with Michael “if a principle or practice helps to improve your efficiency it is no waste”.
-mouneer -
Once the dust has settled and the camps really understand each other, I'm hoping that we'll find that "Lean" and "Agile" are too close to formally join. My guess is that the large product backlog and a proportion of the coding belong in the equivalent of the Toyota Product Development System, rather than their Product System.
More in a
post on my blog. -
It's "Lean" not "Lean Production" for a reason. Lean is about working to reduce the time and energy spent that does not add value to our customer. It's the mantra of Scrum - "inspect and adapt" - continually look at what you're doing, and try to see if there is a better way.
This approach is valuable for, but not limited to production processes. It also applies to new product development, and therefore to software development. Cross-functional teams and emergent design are examples of applying lean philosophy to new product development.
An aspect of Lean altogether missed in this article, is that of knowledge creation and sharing, which brings the supposed world views of Agile and Lean a whole lot closer. The author and readers of this article are recommended to read a little wider on Lean. Mary's list of recommended reading is probably a good place to start: www.poppendieck.com/reference.htm -
I think the author acknowledges, that Agile and Lean principles and practices were created by applying a common philosophy of inspect and adapt. The point being made though is that although the philosophy and values are very similar, the principles and practices are somewhat different. This make sense to me since the two things emerged from different contexts so you wouldn't expect them to be identical. Agile came from Smalltalk and the challenge of Object Orientated programming, and Lean as described in Mary's first book has emerged from manufacturing, and more specifically Toyota and their goal of becoming world leaders at automobile production.
Based on what is in Mary's first book, I don't think she would disagree with what has been said here, but I don't know what she has said since. I also know that Taiichi Ohno also spoke about the pitfalls of blindly following principles and practices out of context. Which is why he chose not to blindly follow the American approach of mass-manufacture which was highly successful in his time.
Paul. -
Why are some people so determined to be divisive? I listened to Dan Jones (wrote The Machine That Changed the World and formulated the concept of Lean from working with Toyota) talk at XPDay last year. He said saw no real difference between Agile and Lean so why do we? Why pick up on one thing that Mary P said about backlogs? She may have written a great book but that does not make her a Caliph or Umma of the practices ergo everyone must obey her every word. If you read The Toyota Way you'll see how frustrated Toyota get by people focusing on particular practices and concepts rather than making the effort to understand the fundamental underlying principles.
As I said recently, Lean will fail for the same reasons Scrum projects do, because people fail to understand the principles and focus on the practices as Dave is doing here. -
Honestly, I completely disagree your statement. If you had done just a little bit of research you would have known that Agile methods have been highly influenced by the TPS right from the beginning. The Poppendiecks were not the first that introduced Lean methods to software development.
See: www.jroller.com/sebastianKuebeck/entry/lean_and... -
Hi Steve,
I think your point about the Toyota Product Development System being more relevant to Software Development than TPS is a good one. I haven't read any primary sources on this, although Mary does mention some of the ideas you refer to in your blog post in her book.
You are probably right, that if you dig deep enough that Lean as applied to new product development rather than production is very similar to Agile, but as you know most people don't bother to dig that deep and just go on the buzzwords and what they've heard second hand from industry leaders.
Paul. -
As an example of what I mean, I read somewhere that Toyota sent a team of Engineers to california for six months to live the high life to get an idea of what Americans saw as luxury living. This was seen as useful background for the development of the Lexus.
Now Engineers living it up in foreign climes sure sound like an ideal example of waste to me :) Yet Toyota saw it as valuable. Should software engineers all be doing the same too? Well I guess it depends on the context. It comes down to giving up on the idea of lifting "best practice" from others and being prepared to do the hard work to find what works in your context for yourself.
Paul. -
Good points Rob. In fact, I would go a step further and say that both lean thinking and agility are far more about state of mind, value systems and establishing a healthy culture. To revisit a point made earlier in the thread, reading about Taiichi Ohno's experiences when he was considering production methods at Toyota, all his ideas were based around open minded thinking - which I find is not such a popular choice these days.
Stepping out of the problem and thinking about it rather than looking for a 'paint by numbers' solution can yield far greater successes - the inspect and adapt mantra cannot be realized without thinking. Far too many ideas are prescriptive - if it doesn't work, change it - so long as values are aligned with success, better ideas can supplant older ones and lead to greater success. -
Hi Andrew,
I agree with everything you say. The one thing I would add though is that Toyota developed the TPS over 3 decades, and despite senior management buy-in, they still made mistakes along the way and had to deal with internal and external resistance as they went. I doubt the software industry has that patience never mind the determination and discipline.
True cultural change is very difficult, and some people are looking for a quick fix. I read Daves blog post as a cautionary warning that blindly adopting Lean practices isn't it.
Going further, If we choose to accept our western organisational culture as part of our context, that is unlikely to change any time soon, then perhaps we need to go about things differently and find a new way, rather than blindly following the Toyota way. Like you say, we need to keep an open mind.
Paul. -
It is always good to see a thought-provoking article. The fact that it seems to have caused some conflict means that Dave has touched on something real. I'm not sure that the points that people make about 'misleading, pointless, and damaging misrepresentation' are quite fair.
As I read it:
In a nutshell: Lean and Agile are grounded in fundamentally different world-views and therefore will inevitably find themselves in opposition on critical points.
The fact that Lean (Production) and Agile (Software Development) have different world views does not mean they intersect. Nor does it mean they are at war. They just mean that they do not fully coincide. (Which I think is fair.)
Furthermore:
In the following paragraphs I will try to show the opposing world-views, illustrate one point of conflict, and then suggest how the two viewpoints might be reconciled.
You'll see that Dave suggests how the world views can be reconciled.
So, instead of sniping, what about addressing the issue - do they have different world views or not? If not, do you think the differences can be ignored or worked with?
Peace. Amr -
Amr, thanks for your comments. If I may amplify:
- Lean values and principles and Agile values and principles are fully complementary.
- The Lean world view - the lens through which Lean values and principles are viewed and interpreted - is grounded in process, specifically production process. This same interpretation is applied to specific practices and specific metrics.
- The Agile world view (excluding, for the moment, Scrum) is grounded in "theory building" and specific practices are interpreted through that lens.
- At the level of values and principles there is little conflict - but, because of the world view lenses, similar values lead to the advocacy of different practices, and occasionally those practices are in apparent conflict.
- My discussion of the product backlog is one example of such conflict.
- not discussed in the article, but Scrum is often misinterpreted in a similar way - as if it was nothing more than an alternate means for imposing Taylorist management principles on software development.
The fact of conflict would be hard to deny - the question of how to resolve that conflict is at issue here. I would assert that it can only be resolved at the level of values and principles. And to accomplish this we need to be consciously aware of how world views (cultures) affect our interpretation and ultimately our expression of those values in concrete practices.
Like many of you who replied to the article - my ultimate goal is to advocate for a thoughtful, fully informed, and deeply reflective software development and the avoidance of doctrinaire, knee-jerk imposition of "by the book" practices. -
I don't think you can apply the adjective blind to "follow the Toyota Way". If part of the way is to examine what you do and adapt, then what pls give an example of how to do that blindly.
Jeff -
May I suggest that it is not that hard to deny the conflict, when what was presented was highly hypothetical, and I have never seen any real conflict myself.
Your suggested solution to the conflict implies that both Lean and Agile are self-contradictory. If the solution is as you state -
"Both Lean and Agile must stop applying, in a literal and rote manner, the tools and practices." and both Agile and Lean are in favor of inpecting and adapting, as their guiding principles suggest that they should, then they cannot be applying "the tools and practices" in a "literal and rote manner" unless they are not following their own principles.
I can understand the differences highlighted in the article, Although I think event these hypothetical differences require a narrow interpretation of each Agile and Lean, but I think to argue that these differences are causing real problems in software development environments today requires a little more evidence before the reader can be asked to accept the argument given.
Jeff -
I think product backlog was a bad example to make your point. Actually I don't see product backlog as a main practice of agile methodologies.
I already worked at several projects with huge backlog lists that worth noting. It's even painful to remember when Scrum Master insists in review all those cards (that God know when one of those will come into current sprint) every meeting regardless the bored team views no value on that. The client asking us to remove half of those cards because they don't make sense anymore was only a matter of time.
Mary and Tom made their point quite clear when they said that there are differences between Lean Production and Lean Software Development. The key was the principles, tools and process can be adapted.
Guilherme Lopes -
Actually I don't see product backlog as a main practice of agile methodologies.
So what mechanism(s) do you use to understand the problem at hand and have a conversation with your customers? Do you keep a record? -
I recently published a "survey" which presents the "Programming as Theory Building" paper (Naur 1985) and later comments from papers and web. dl.getdropbox.com/u/502901/naur.pdf. It includes the relevant excerpt from Cockburn 2006.
Abstract: The present survey republishes major articles, reprints leading discussion, and collects erudite comment on the theory-building view of software programming. -
Whenever I read blogs like this I often wonder who the target audience is. From my perspective, this is way to theoretical for anyone in the trenches to get any real benefit from this argument. My sense is this is just a way to stir up conversation amongst the purists. I don't believe anyone doing actual software development really cares - sorry. My opinion is that Lean has some great concepts that really work well with Agile methods and I have written about this on my own blog at blog.agilebuddy.com
This was way over the top for me
My 2 cents
Jack -
So what mechanism(s) do you use to understand the problem at hand and have a conversation with your customers? Do you keep a record?
We all (client and developers) have a shared vision about the product. We discuss it several times in week. The client is really close to us, actually We have lunch together every day.
But definitions, written on ink and paper, issue tracker or whatever, are only made for the very next iteration.
We don't need to keep a record. The important features will keeps coming back, naturally. -
Can you see by the number of comments just how bored software professionals get around Valentine day? They will do anything, even post very long comments, to flee their romantic duties.
Anyway.
Just wanted to make a point here, that we have ALL made this mistake. There is no ONE "Lean". The TPS (Toyota Production Systems) and the TPDS (Toyota Product Development System) are TWO entirely distinct systems. Just to cite Jeffrey Liker: "Most Toyota Product Development managers claimed they had very limited knowledge of TPS, and Toyota engineers did not see TPS as the launching point for lean processes in product development."
And it should be noted that the Poppendiecks published a second book, that underlined the same mistake at its very beginning. If someone is considering reading about Lean Software Development, get the latest book (2007)!
Make love, not war. -
Large queues in systems exhibit certain characteristics (see Little’s law for example). The optimal size for a given queue depends entirely on context so an assertion that a large backlog (a form of queue) is either good or bad absent of a project context is fallacious. In general cost and average time to produce value increases as a queue grows however; if the net value created for the customer is increased with a larger queue then a larger queue is preferable. In other words, big backlogs are bad except when they’re not. The singular goal of lean is to increase value. Fixating on one principle or constraint (i.e. cost increases with queue size) without balancing competing constraints (i.e. more design variation / theories can increase value) is a misapplication of lean.
Despite its origins, the core principles of lean thinking are orthogonal to the specific practices that have evolved. Lean principles gave rise to lean production; they’ve also given rise to lean development practices. In production variation usually decreases value. In development, variation often increases value. ‘Lean’ does not wear “production glasses”; far too often the people trying to apply lean principles and practices do.
If we’re going to advance the state of the art and help our teams we need to focus less on practices and more on the underlying principles. Lean development is no more about ‘small backlogs’ than Agile is about ‘no documentation’. Lean and agile principles are entirely complimentary. Often, the practices evolved are not. What we need are more practitioners and coaches in the field who deeply understand the underlying principles of lean and agile, who live and breathe ‘innovate, inspect, and adapt’, and aren’t wedded to single-source methodology dogma. That might not sell as many books, but it sure as heck will increase value for our customers.
I greatly admire and appreciate the Poppendieck's contribution to improving software development but to gain a fundamental understanding of Lean you’re going to have to read more deeply and broadly than their two lean books. Don Reinertsen has an illuminating new book coming out in May (The Principles of Product Development Flow: Second Generation Lean Product Development) that I strongly encourage anybody trying to get a handle on applying lean in a development context to read.
Jay Packlick -
I like the idea of Programming as Theory Building.
In lean, learning starts with building a theory and then testing it, finding where the theory does not hold up, modifying it, and so on forever. It's pretty much a constant Plan-Do-Check-Act cycle. You do this with the product (software). You do this with the development process (constant improvement).
So I suggest that the way to handle a backlog is to build a theory on how long it should be in your environment, create a theory on what the results of that length would be. Devise a way to test the theory. Run the experiments. Learn.
Mary Poppendieck -
Backlogs as officially described are just a list of items. These flat backlogs, as Jeff Patton calls them, are not really that effective as models for the theory of the product or system. In essence, they are just a list of items to do.
A story map, a Parking Lot diagram, process map, etc. serve the role of theory model better than a backlog.
The other important point, which is really only apparent when we look at the detailed day-to-day issues of software development (and even product development), is that there is still a significant part of what we do that is more repetitive than not. Especially when we realise that most software development is maintenance, not new concept development.
Of course, software development is not the same as what you see in any particular factory. But that's why the essential part "Lean" is not the particular solutions designed for a particular context. The essential part is the worldview: respect for people, continuous improvement, problem solving, focus on the customer, etc. It's a very common message in the Lean community about not just inserting some off-the-shelf solution into your own situation but actually thinking about what's going on.
Someone with a very tool-focused worldview will believe that Lean and Agile are incompatible. Someone with a values-focused worldview will see that the thinking approach is very compatible. I'd even argue that people with a tool-focused worldview don't really get what was eventually called Agile anyway. -
I really like this metaphor - enough to quote it here<blockquoten my head a see a tree where the trunk is built from the goals or desired benefits that drive the system; big branches are users; the small branches and twigs are the capabilities they need; then finally the leaves are the user stories small enough to place into development iterations."
"After all that work, after establishing all that shared understanding I feel like we pull all the leaves off the tree and load them into a leaf bag - then cut down the tree."
"That's what a flat backlog is to me. A bag of context-free mulch.">I agree with the sentiment and agree that the "flat product backlog" is not the best tool for advancing theory. Fortunately, the backlog is not the only agile practice that promotes theory building - stand-up meetings, big visible charts, automated tests, collective code ownership, wall-to-wall whiteboards, retrospectives, planning games, etc. make their contributions. In concert they may or may not be enough to fully support theory building and any one of them could certainly be improved.
Unfortunately, very few people look at these practices and see their contribution to theory building - just as very few see their relation to the underlying values and principles of agile. The most controversial accusation I made in the original article was implicit: practitioners of agile, and lean, do not really understand WHY they are doing what they are doing. Because of that they tend to misuse, misinterpret, or misapply the practices.
A product backlog is one example. As a coach, not a theorist or academic, I have very rarely seen teams using the product backlog as anything other than an inventory of to do items. (Until I coached them differently, of course. [smile]) If this is the extent of your understanding of why a backlog exists, it will be hard not to see it as waste.
A similar error occurs with either and both of the Toyota models - the superficial understanding of the practices - i.e. an understanding that is not deeply grounded in an understanding of the underlying principles and values - leads to the same misinterpretations. In the case of Lean, it is really easy to make the mistake of interpreting the practices as being solely related to the idea of production - because the production model is the de facto culture of software development.
davew
<//blockquote> -
sorry about the formatting errors in the previous reply - the system gave me an error message when trying to preview and then posted when I acknowledged the error - not expected behavior.
-
Jay,
Very well said - I have very little to add after reading your comment. And, thanks for the reference to the upcoming book.
Most of us have accepted the Poppendieck's books as the launchpads of our journey and moved on. Theory building, without a doubt, happens all the time when you are doing Lean properly. Lean is all in the "thinking" - in fact some of the best books about Lean are those address the thinking - achieving deeper undrestanding, model building - theory building.
Dave,
if you read a little bit more literature about the "other" TPS - Toyota Product-development System... you will find that the theory is in violent agreement with what you have to say.
All,
A wonderful book that might help address some of the misgivings of Lean as applied to software development is : www.lean.org/Bookstore/ProductDetails.cfm?Selec... . It is one of the most "highlighted" of all lean books I have in my stack. -
I'm not following why backlogs are essential. We were convinced (partly by Mary and Tom) that our backlog was silly overhead. We only keep enough in it now for the next couple or three sprints and this has been working fine for a year or two. I feel that we have adjusted our "agile" with some "lean" principles and it's working for us.
-
These stories in the bscklog are isomorphic with the long list of requirements in formal IT contracts. When there are six or ten of them that all the customers are agreed on, this will work well. When there are fifty or two hundred, then they are likely just collected lists of wishes which have not been vetted by the other customers.
Don't accept them. Treat them as hints and use the five (or more) whys to find the business purpose of each. Take the five or ten ultimate business purposes as your goals and decide how to quantify progress toward each goal. These might include saving time or other resource, increasing revenue, decreasing complaints, or any rational business need.
Now engineer some approaches toward satisfying each goal. Discuss these with the product owner or better, the customers. Build them into stories. Estimate the impact that each story will have on each goal and choose the greatest impact lowest effort stories to address first.
Everyone will be happier with the results and every implemented feature will be used. And you can quit when the expected progress toward the goals of any remaining story is worth less than the cost of implementing it. -
In a forum debate that included several of the original members of the agile alliance, they agreed that lean thinking had not been an explicit influence on their thinking at the time of the manifesto. Sure, we can look back today and see the parallels and the synergy, but let's not be revisionist.
32 comments
Watch Thread Reply