Tom Gilb and Lindsey Brodie have written an article that suggests that Agile methods have a major weakness - that of lack of quantification. They argue that all qualities can be expressed quantitatively and present a new process, PLanguage, which looks very much like Scrum with an explicit measurement step. Are they right? Are Agile methods such as Scrum and XP in need of explicit measurement?
In, What's Wrong With Agile Methods: Some Principles and Values to Encourage Quantification, Gilb and Brodie suggest:
Agile Software Methods (Agile Alliance 2006) have insufficient focus on quantified performance levels (that is, metrics stating the required qualities, resource savings and workload capacities) of the software being developed. Specifically, there is often no quantification of the main reasons why a project was funded (that is, metrics stating the required business benefits, such as business advancement, better quality of service and financial savings). This means projects cannot directly control the delivery of benefits to users and stakeholders. In turn, a consequence of this is that projects cannot really control the corresponding costs of getting the main benefits. In other words, if you don’t estimate quantified requirements, then you won’t be able to get a realistic budget for achieving them.
This claim will, at best, not impress those with any practical experience with Agile methods, because at their core, Agile methods are Inspect and Adapt methods. Inspection implies measurement. However, there is very little explicit guidance in how to measure and what to measure. Gilb and Brodie do not give explicit guidance on how to measure, but they do give a running example of quantification of one requirement.
On the other hand, InfoQ's own Deborah Hartmann, has co-written Appropriate Agile Measurement: Using Metrics and Diagnostics to Deliver Business Value with Robin Dymond, which suggests measurement is important and a good measurement candidate:
- Affirms and reinforces Lean and Agile principles.
- Measures outcome, not output.
- Follow trends, not numbers.
- Belongs to a small set of metrics and diagnostics.
- Is easy to collect.
- Reveals, rather than conceals, its context and significant variables.
- Provides fuel for meaningful conversation.
- Provides feedback on a frequent and regular basis.
- May measure Value (Product) or Process.
- Encourages "good-enough" quality.
For each business value come up with at least one way that you can take a measurement of progress made. That is, if you are to implement a practice to improve a particular business value you will need to take a periodic reading to verify that the practice is working. This does not have to be quantitative. It may be qualitative in nature. Make it as simple as possible. For example, if you want to take a measurement to reduce cost, a simple (and rough) reading would be the number of hours put in for a major release.
We then must periodically measure our progress to decide if the practice we have adopted are helping us meet our goals.
It is worthwhile to note that none of these sources tell us exactly how to make the measurements and what those measurements should be. This is, of course, because it depends! It is also thought-provoking that Gilb and Brodie see measurements as missing from Agile, while many of us who are practicing see it as an integral part of the way we work. Is this similar to the outside world seeing Agile as 'cowboy-coding' while we know we are disciplined?
Community comments
Scrum is certainly measured
by Angus McDonald,
Why not try mingle
by anchuan qian,
Re: Why not try mingle
by Amr Elssamadisy,
Re: Why not try mingle
by anchuan qian,
Who does it better?
by Jim Standley,
Re: Who does it better?
by Robert Merrill,
An ounce of software
by Robert Merrill,
Re: An ounce of software
by Sidu Ponnppa Chonira Kariappa,
Any practical experience?
by pjbxkg5wd46vhq8 Temp,
Re: Any practical experience?
by Chris Matts,
Re: Any practical experience?
by Amr Elssamadisy,
Re: Any practical experience?
by Deborah (Hartmann) Preuss,
Re: Any practical experience?
by Deborah (Hartmann) Preuss,
Re: Any practical experience?
by pjbxkg5wd46vhq8 Temp,
Scrum is certainly measured
by Angus McDonald,
Your message is awaiting moderation. Thank you for participating in the discussion.
Don't we measure things when we track velocity for our projects, the number of story points in a sprint (planned and actual) and the percentage of a developer's time available for the sprint (focus)?
However they may be right, a more rigorous software engineering approach might be the best one for the creation of "industrial products". Agile is good for lots of projects, but not every one possible.
On the other hand, the business must have a good idea what the expected benefit is from a particular sprint (preferably from each story) and they should measure the effect that the sprint has towards that benefit. But that is something I would regard as explicitly outside the development team's area of control (although they may be involved in providing the raw data towards measuring it).
The whole point of the TQM revolution was that quality is not something that gets left to last, then measured. By making continuous integration and automated testing/builds part of our daily practices we already make a BIG step in the right direction to ensure good quality.
Why not try mingle
by anchuan qian,
Your message is awaiting moderation. Thank you for participating in the discussion.
Why not try mingle?
studios.thoughtworks.com/mingle-project-intelli...
Re: Why not try mingle
by Amr Elssamadisy,
Your message is awaiting moderation. Thank you for participating in the discussion.
LoL,
Ok, I'll bite...
What does Mingle have to do with this? Or is that an advertisement for TW's new toy?
Remember- People and Interactions over Process and Tools
How can a tool help solve the problem of what to measure and how to go about doing so?
Who does it better?
by Jim Standley,
Your message is awaiting moderation. Thank you for participating in the discussion.
I've been at this business just shy of 30 years and have yet to see a formal, published study any time after project completion to say the ROI that we projected was met or not. Is that the kind of measurement people are after? Anybody here do such studies regularly?
An ounce of software
by Robert Merrill,
Your message is awaiting moderation. Thank you for participating in the discussion.
Maybe outsiders see the lack of OUTWARD-FACING metrics, that can be compared across projects or even industry-wide. Agile thought leaders seem metrics-averse, maybe out of reaction to the way metrics have been used (and abused) in the past.
As I see it, the basic four, productivity (X/effort), velocity (X/time), quality (defect/X), and value (benefit/X) still apply, and within a project, agile teams track and respond to them, in their own effective ways (measured velocity, continuous integration and automated tests, planning game). But when you need to compare projects, you need a more portable X, or what I call "an ounce of software."
I used to do project set-up for a contracting shop that used agile methods. We needed a decent estimate before we even got the chance to establish a velocity. That estimate needed to be clearly traceable to the initial requirements, to help the client think "scope-to-budget" from the start. The estimate also needed to be tied to actual results on other projects, so it couldn't be pressured down.
Based on these requirements, I introduced Function Points. Usually dismissed with a sniff by agilists, they did just what we needed, and didn't interfere with people and interactions at all. They also helped our initial story development by revealing gaps and implied stories.
www.ufunctional.com
Re: An ounce of software
by Sidu Ponnppa Chonira Kariappa,
Your message is awaiting moderation. Thank you for participating in the discussion.
It is impossible today to say that project X with design Y produced a cost savings of $Z and if design A had been used the cost saving would have been $B.
Quality of software is still subjective and depends on context (heavily meta-programmatic implementation may be elegant is one case but not when resources are restricted) and comparing one project with another is again impossible. Give this, metrics are of no use in comparing the performance or value of different projects.
And within a single project, all that one can extract is simple trends (not that these are not useful - in fact trends are very useful). But trying to pin down something subjective and unquantifiable- I'm not sure how useful and accurate that would be.
Re: Who does it better?
by Robert Merrill,
Your message is awaiting moderation. Thank you for participating in the discussion.
Many eCommerce operations do this, because the software sits astride a revenue stream.
The folks at TDS Telecom appear to be doing this with their internal apps, too.
I don't see as much of it as I like, either, but I suspect that this is going on anywhere that IT is viewed as something more than a cost to be minimized.
It's essential if you want to see the whole picture, rather than just optimize one little piece. Consider two projects. One went over budget by 2x, but delivered 4x the expected business benefit. The other came in on budget, but delivered 1/2 the expected business benefit. Which was the success?
Re: Why not try mingle
by anchuan qian,
Your message is awaiting moderation. Thank you for participating in the discussion.
Amr Elssamadisy,Good question.
I totally agree "People and Interactions over Process and Tools".But the Tools can do many things that people can not do.So,we deliver software to our clients.
You know, there is no standard to measure the software development.It is different between different team and different project.So you must know exact how you did before.The excellent tool should holds the real project state in real time.At same time, it should be simple and easy to use, Nobody are interested in bloated applications.
If try mingle or not,It is you privilege. I just read this article and think Mingle supports this process.
Any practical experience?
by pjbxkg5wd46vhq8 Temp,
Your message is awaiting moderation. Thank you for participating in the discussion.
"This claim will, at best, not impress those with any practical experience with Agile methods"
That's the problem! As much as I appreciate most of Gilb's work when he writes/talks about Agile he clearly shows a non-practical understading of it. He has clearly never tried it and this makes me wonder: does he have any practical experience of all the things he wrote/spoke about in all these years? Or he just talks about theory and has never actually done anything concrete?
Re: Any practical experience?
by Chris Matts,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi pjbxkg5wd46vhq8 Temp (Nice name? reminds me of the sprite in Alan Moore's "Death of Superman")
Last week I was lucky enough to attend mini conference with Tom Gilb and a number of people who have been working with his Evo method for decades.
At one point someone, not Tom, mentioned that a project did not even have metrics on usability. The group laughed... I for one am curious and would like to bring the Evo community into the mainstream to find out more. I suspect that they do have experience of metrics, but their experience is at a much higher level than we can currently conceive.
We can chose to ignore them, or we can listen. The first option provide no opportunity for learning. If they do not have any useful experience then all we sacrifice is a few hours of our time. If they do, we can save years of effort and learning.
Chris
Re: Any practical experience?
by Amr Elssamadisy,
Your message is awaiting moderation. Thank you for participating in the discussion.
Chris,
First of all let me agree with you that it would be great to bring in the Evo folks for their comments and wisdom.
On the other hand - saying that agile methods do not measure is incorrect and naive. We measure all the time. XP and Scrum are all about inspect-and-adopt cycles.
Now, the fact that we don't necessarily measure 'business value' is true to a large extent. (see today's news item: www.infoq.com/news/2007/07/IT_Business_Gap ) But the paper referenced here does some 'hand-waving' about what business value is and how to measure it specifically.
My 2 piasters,
Amr
Re: Any practical experience?
by Deborah (Hartmann) Preuss,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hello Mr. or Ms. "Temp". In the interest of transparent discourse, I'd encourage you to use your real name in your profile here. Thanks!
Re: Any practical experience?
by pjbxkg5wd46vhq8 Temp,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi! I understand but I'm not going to, too many bad experiences in the past. If it's a problem (which, again, I'd understand) you should probably make it clear in the registration form that you don't want nicknames, fakenames or whatever. Regards
Re: Any practical experience?
by Deborah (Hartmann) Preuss,
Your message is awaiting moderation. Thank you for participating in the discussion.
Chris - will Evo folks be at Agile2007? Maybe we should try to get an adhoc metric session going?