Agile Measurement - A Missing Practice?

Posted by Amr Elssamadisy on Jul 04, 2007 03:18 PM

Community

Agile

Topics

Methodologies ,

Agile Techniques

Tags

Scrum ,

XP ,

Adoption ,

Lean

Tom Gilb and Lindsey Brodie have written an article that suggests that Agile methods have a major weakness - that of lack of quantification. They argue that all qualities can be expressed quantitatively and present a new process, PLanguage, which looks very much like Scrum with an explicit measurement step. Are they right? Are Agile methods such as Scrum and XP in need of explicit measurement?

In, What's Wrong With Agile Methods: Some Principles and Values to Encourage Quantification, Gilb and Brodie suggest:

Agile Software Methods (Agile Alliance 2006) have insufficient focus on quantified performance levels (that is, metrics stating the required qualities, resource savings and workload capacities) of the software being developed. Specifically, there is often no quantification of the main reasons why a project was funded (that is, metrics stating the required business benefits, such as business advancement, better quality of service and financial savings). This means projects cannot directly control the delivery of benefits to users and stakeholders. In turn, a consequence of this is that projects cannot really control the corresponding costs of getting the main benefits. In other words, if you don’t estimate quantified requirements, then you won’t be able to get a realistic budget for achieving them.

This claim will, at best, not impress those with any practical experience with Agile methods, because at their core, Agile methods are Inspect and Adapt methods. Inspection implies measurement. However, there is very little explicit guidance in how to measure and what to measure. Gilb and Brodie do not give explicit guidance on how to measure, but they do give a running example of quantification of one requirement.

On the other hand, InfoQ's own Deborah Hartmann, has co-written Appropriate Agile Measurement: Using Metrics and Diagnostics to Deliver Business Value with Robin Dymond, which suggests measurement is important and a good measurement candidate:

1. Affirms and reinforces Lean and Agile principles.
2. Measures outcome, not output.
3. Follow trends, not numbers.
4. Belongs to a small set of metrics and diagnostics.
5. Is easy to collect.
6. Reveals, rather than conceals, its context and significant variables.
7. Provides fuel for meaningful conversation.
8. Provides feedback on a frequent and regular basis.
9. May measure Value (Product) or Process.
10. Encourages "good-enough" quality.

This sentiment, that we must continuously measure and respond to what we find, is true even in the meta-process sense. In Patterns of Agile Practice Adoption: The Technical Cluster, we are told that to decide what Agile practices are needed, we must first focus on business values that we want to affect by adopting agile practices, and then:

For each business value come up with at least one way that you can take a measurement of progress made. That is, if you are to implement a practice to improve a particular business value you will need to take a periodic reading to verify that the practice is working. This does not have to be quantitative. It may be qualitative in nature. Make it as simple as possible. For example, if you want to take a measurement to reduce cost, a simple (and rough) reading would be the number of hours put in for a major release.

We then must periodically measure our progress to decide if the practice we have adopted are helping us meet our goals.

It is worthwhile to note that none of these sources tell us exactly how to make the measurements and what those measurements should be. This is, of course, because it depends! It is also thought-provoking that Gilb and Brodie see measurements as missing from Agile, while many of us who are practicing see it as an integral part of the way we work. Is this similar to the outside world seeing Agile as 'cowboy-coding' while we know we are disciplined?

1. Back to top

   Scrum is certainly measured

   Jul 4, 2007 6:37 PM by Angus McDonald

   Don't we measure things when we track velocity for our projects, the number of story points in a sprint (planned and actual) and the percentage of a developer's time available for the sprint (focus)? However they may be right, a more rigorous software engineering approach might be the best one for the creation of "industrial products". Agile is good for lots of projects, but not every one possible. On the other hand, the business must have a good idea what the expected benefit is from a particular sprint (preferably from each story) and they should measure the effect that the sprint has towards that benefit. But that is something I would regard as explicitly outside the development team's area of control (although they may be involved in providing the raw data towards measuring it). The whole point of the TQM revolution was that quality is not something that gets left to last, then measured. By making continuous integration and automated testing/builds part of our daily practices we already make a BIG step in the right direction to ensure good quality.

   Reply

2. Back to top

   Why not try mingle

   Jul 4, 2007 10:47 PM by qian anchuan

   Why not try mingle? http://studios.thoughtworks.com/mingle-project-intelligence

   Reply

3. Back to top

   Re: Why not try mingle

   Jul 5, 2007 8:58 AM by Amr Elssamadisy

   LoL, Ok, I'll bite... What does Mingle have to do with this? Or is that an advertisement for TW's new toy? Remember- People and Interactions over Process and Tools How can a tool help solve the problem of what to measure and how to go about doing so?

   Reply

4. Back to top

   Who does it better?

   Jul 5, 2007 10:23 AM by Jim Standley

   I've been at this business just shy of 30 years and have yet to see a formal, published study any time after project completion to say the ROI that we projected was met or not. Is that the kind of measurement people are after? Anybody here do such studies regularly?

   Reply

5. Back to top

   An ounce of software

   Jul 5, 2007 10:29 AM by Robert Merrill

   Maybe outsiders see the lack of OUTWARD-FACING metrics, that can be compared across projects or even industry-wide. Agile thought leaders seem metrics-averse, maybe out of reaction to the way metrics have been used (and abused) in the past. As I see it, the basic four, productivity (X/effort), velocity (X/time), quality (defect/X), and value (benefit/X) still apply, and within a project, agile teams track and respond to them, in their own effective ways (measured velocity, continuous integration and automated tests, planning game). But when you need to compare projects, you need a more portable X, or what I call "an ounce of software." I used to do project set-up for a contracting shop that used agile methods. We needed a decent estimate before we even got the chance to establish a velocity. That estimate needed to be clearly traceable to the initial requirements, to help the client think "scope-to-budget" from the start. The estimate also needed to be tied to actual results on other projects, so it couldn't be pressured down. Based on these requirements, I introduced Function Points. Usually dismissed with a sniff by agilists, they did just what we needed, and didn't interfere with people and interactions at all. They also helped our initial story development by revealing gaps and implied stories. www.ufunctional.com

   Reply

6. Back to top

   Re: An ounce of software

   Jul 5, 2007 10:53 AM by Sidu Ponnppa Chonira Kariappa

   It is impossible today to say that project X with design Y produced a cost savings of $Z and if design A had been used the cost saving would have been $B. Quality of software is still subjective and depends on context (heavily meta-programmatic implementation may be elegant is one case but not when resources are restricted) and comparing one project with another is again impossible. Give this, metrics are of no use in comparing the performance or value of different projects. And within a single project, all that one can extract is simple trends (not that these are not useful - in fact trends are very useful). But trying to pin down something subjective and unquantifiable- I'm not sure how useful and accurate that would be.

   Reply

7. Back to top

   Re: Who does it better?

   Jul 5, 2007 7:05 PM by Robert Merrill

   Many eCommerce operations do this, because the software sits astride a revenue stream. The folks at TDS Telecom appear to be doing this with their internal apps, too. I don't see as much of it as I like, either, but I suspect that this is going on anywhere that IT is viewed as something more than a cost to be minimized. It's essential if you want to see the whole picture, rather than just optimize one little piece. Consider two projects. One went over budget by 2x, but delivered 4x the expected business benefit. The other came in on budget, but delivered 1/2 the expected business benefit. Which was the success?

   Reply

8. Back to top

   Re: Why not try mingle

   Jul 5, 2007 7:43 PM by qian anchuan

   Amr Elssamadisy,Good question. I totally agree "People and Interactions over Process and Tools".But the Tools can do many things that people can not do.So,we deliver software to our clients. You know, there is no standard to measure the software development.It is different between different team and different project.So you must know exact how you did before.The excellent tool should holds the real project state in real time.At same time, it should be simple and easy to use, Nobody are interested in bloated applications. If try mingle or not,It is you privilege. I just read this article and think Mingle supports this process.

   Reply

9. Back to top

   Any practical experience?

   Jul 10, 2007 9:21 AM by pjbxkg5wd46vhq8 Temp

   "This claim will, at best, not impress those with any practical experience with Agile methods" That's the problem! As much as I appreciate most of Gilb's work when he writes/talks about Agile he clearly shows a non-practical understading of it. He has clearly never tried it and this makes me wonder: does he have any practical experience of all the things he wrote/spoke about in all these years? Or he just talks about theory and has never actually done anything concrete?

   Reply

10. Back to top

    Re: Any practical experience?

    Jul 10, 2007 12:50 PM by Chris Matts

    Hi pjbxkg5wd46vhq8 Temp (Nice name? reminds me of the sprite in Alan Moore's "Death of Superman") Last week I was lucky enough to attend mini conference with Tom Gilb and a number of people who have been working with his Evo method for decades. At one point someone, not Tom, mentioned that a project did not even have metrics on usability. The group laughed... I for one am curious and would like to bring the Evo community into the mainstream to find out more. I suspect that they do have experience of metrics, but their experience is at a much higher level than we can currently conceive. We can chose to ignore them, or we can listen. The first option provide no opportunity for learning. If they do not have any useful experience then all we sacrifice is a few hours of our time. If they do, we can save years of effort and learning. Chris

    Reply

11. Back to top

    Re: Any practical experience?

    Jul 10, 2007 3:32 PM by Amr Elssamadisy

    Chris, First of all let me agree with you that it would be great to bring in the Evo folks for their comments and wisdom. On the other hand - saying that agile methods do not measure is incorrect and naive. We measure all the time. XP and Scrum are all about inspect-and-adopt cycles. Now, the fact that we don't necessarily measure 'business value' is true to a large extent. (see today's news item: http://www.infoq.com/news/2007/07/IT_Business_Gap ) But the paper referenced here does some 'hand-waving' about what business value is and how to measure it specifically. My 2 piasters, Amr

    Reply

12. Back to top

    Re: Any practical experience?

    Jul 11, 2007 4:54 AM by Deborah Hartmann

    Hello Mr. or Ms. "Temp". In the interest of transparent discourse, I'd encourage you to use your real name in your profile here. Thanks!

    Reply

13. Back to top

    Re: Any practical experience?

    Jul 11, 2007 8:03 AM by pjbxkg5wd46vhq8 Temp

    Hi! I understand but I'm not going to, too many bad experiences in the past. If it's a problem (which, again, I'd understand) you should probably make it clear in the registration form that you don't want nicknames, fakenames or whatever. Regards

    Reply

14. Back to top

    Re: Any practical experience?

    Jul 20, 2007 9:34 PM by Deborah Hartmann

    Chris - will Evo folks be at Agile2007? Maybe we should try to get an adhoc metric session going?

    Reply

• Older >