Debate: Agile Transition Success Rates, Help or Harm?
Nearly 250 posts have been registered in response to Niraj Khanna's question on the extremeprogramming yahoo list:
I have been searching for the success or failure rate of agile transition attempts, but have come up relatively empty. I would define success as:Much of the early discussion equated to a debate regarding whether a search for such metrics might turn up less than flattering results. This debate was somewhat rooted in a suggestion that the agile community may fear what such results might show, as described by Chris Wheeler:
a) The transition met the goals/improvements the customer or company was looking for. The goals/improvements could be anything from bottom line profit to productivity to fewer defects and shorter project timelines of delivery.
b) If the improvements the agile coach (3rd party or internal) promised to the company came to fruition.
c) If there were any pleasant corollary improvements that weren't expected.
Failure would be simply defined as: The time and money spent on the transition were either not worth the result. In the worst cases, the development organization actually lost quality, speed or even customers due to the transition attempt.
For years, the concept of using metrics to communicate to executives (or to anyone) was resisted by the Agile community ... I don't think we lack the knowledge so much as we are afraid that when we look at the numbers they won't paint the same great story that, well, stories show.Many quickly advanced beyond this suggestion, agreeing that fear of the numbers are not a problem. Helping to put an objective and positive lid on the "are we afraid of these numbers" discussion, Kent Beck said the following:
Absolute openness about the costs, pains, risks, and rewards of XP is the right thing to do. It's a leadership position and it has a chance of working. I'm afraid that continuing to try to tell an attractive story will lead this community to become a cult of irrelevant technical competence.Much of the debate soon then focused on questioning the verifiability of such statistics, many taking the stance that trying to gather statistics such as these might have a low likelihood of being reliable, as described Steve Gordon:
I am against processing those case studies into numbers, because those numbers would only be valid if the case studies we have are a random sample. They are not a random sample, so those numbers do not represent reality whether or not those numbers serve our interests.Max Guernsey added warning that statistics, even if accurate, may be harmful due to likely misinterpretation:
Statistics that come from "historical surveys" are really not very effective tools because the filter of interpretation plays too large a role. They are given the same kind of credibility as real science but they are missing one key element: verifiability. There is always an element of the interpreters fantasy mixed in with the "results"...Chris Wheeler largely leads the counter to these points, asserting that such data could be utilized effectively if used not as a scientific measurement but rather as helpful tool for executives and managers to get a good "gut feel":
None of this says 'Yes, you will be 100% successful, or 75% successful'. It will say '80% of companies in the auto industry spent between $1M and $1.5M to transition. 50% of Agile programs paid all transition costs after 3 years, 40% after 4 years, and 10% after 5 years.. After 5 years, 60% of those companies are still using agile, 20% are using some agile techniques, and 20% have abandoned agile all together.'The thread's original poster, Niraj Khanna, added this to support Wheeler's stance:
Now, imagine I, as a CIO had that information at available to me! I could use that information to support a decision in the context of my own business. Perhaps 3 years is too long to realize returns, or maybe it's about right and I'm willing to live with the risk of going longer than 3 years. Or maybe I'm not willing to live with that risk - perhaps 3 years is way too long to get my 1.5M back, and maybe it's about right. Maybe $1M is about right, but I can't risk seeing my entire investment dissolve after 5 years.
I think that a broad indicator like a success rate would serve some meaningful purpose, both for sales and for those of us involved in transitions. I like to think of it in parallel to what the Dow Jones composite index is to the stock market. The indicator itself doesn't tell you why the market dropped or rose 200 pts. The purpose it serves is merely informing investors how the broader market did that day. It's up to the individual investor to find out why.One subject area nearly all of the thread's contributors seemed to agree on was the value of collecting real stories of success and failure, asserting this information above all else can serve to provide newcomers useful information about what they might expect when deciding to take on a transition to agile. Perhaps most newsworthy out of all of this was the initiation by Kent Beck of a new wiki site dedicated to an effort to collect such stories from the community:
In the spirit of data over speculation and the spirit of crowd-sourcing over heroic action, I've created a wiki to gather information about transitions: http://xpprojects.wikispaces.com/. Please consider taking ten minutes to record your organization's experience there.Ron Jeffries agrees, simply stating:
Given the number of people on this list, we could, should we choose to, quickly have hundreds of stories from which to learn. This learning is a prerequisite to being able to say, twenty years from now, that software development is an exemplar of quality and value.
That would be most excellent.As you might guess, these are only select highlights of the super discussion still occurring in this XP Yahoo list thread. If you're interested in more, check out the continuing thread for yourself here.
Lean transitions apparently have a 90% failure rate
So the question is why? What is needed to figure out why? What is needed to figure out how to get better?
Part of that is to know when attempts fail and when they succeed. Hiding from information, even if imperfect is not the path to improvement.
If you're going to measure agile transition metrics...
The pros also have trouble measuring this
Of course, we're expecting another report late in 2008...
Do you think this a good idea for the alliance? Do you have some other good examples?
Re: Story Repository
Super idea. I'd love to see the great new way we're selecting the content for this year's Agile Conf -- "post, share, and collaborate" -- become an ongoing way for us all to continue to "post, share, and collaborate".
Let me know if if and how I'd be able to help.