Agile Measurement - A Missing Practice?

Tom Gilb and Lindsey Brodie have written an article that suggests that Agile methods have a major weakness - that of lack of quantification. They argue that all qualities can be expressed quantitatively and present a new process, PLanguage, which looks very much like Scrum with an explicit measurement step. Are they right? Are Agile methods such as Scrum and XP in need of explicit measurement?

In, What's Wrong With Agile Methods: Some Principles and Values to Encourage Quantification, Gilb and Brodie suggest:

Agile Software Methods (Agile Alliance 2006) have insufficient focus on quantified performance levels (that is, metrics stating the required qualities, resource savings and workload capacities) of the software being developed. Specifically, there is often no quantification of the main reasons why a project was funded (that is, metrics stating the required business benefits, such as business advancement, better quality of service and financial savings). This means projects cannot directly control the delivery of benefits to users and stakeholders. In turn, a consequence of this is that projects cannot really control the corresponding costs of getting the main benefits. In other words, if you don’t estimate quantified requirements, then you won’t be able to get a realistic budget for achieving them.

This claim will, at best, not impress those with any practical experience with Agile methods, because at their core, Agile methods are Inspect and Adapt methods. Inspection implies measurement. However, there is very little explicit guidance in how to measure and what to measure. Gilb and Brodie do not give explicit guidance on how to measure, but they do give a running example of quantification of one requirement.

On the other hand, InfoQ's own Deborah Hartmann, has co-written Appropriate Agile Measurement: Using Metrics and Diagnostics to Deliver Business Value with Robin Dymond, which suggests measurement is important and a good measurement candidate:

1. Affirms and reinforces Lean and Agile principles.
2. Measures outcome, not output.
3. Follow trends, not numbers.
4. Belongs to a small set of metrics and diagnostics.
5. Is easy to collect.
6. Reveals, rather than conceals, its context and significant variables.
7. Provides fuel for meaningful conversation.
8. Provides feedback on a frequent and regular basis.
9. May measure Value (Product) or Process.
10. Encourages "good-enough" quality.

This sentiment, that we must continuously measure and respond to what we find, is true even in the meta-process sense. In Patterns of Agile Practice Adoption: The Technical Cluster, we are told that to decide what Agile practices are needed, we must first focus on business values that we want to affect by adopting agile practices, and then:

For each business value come up with at least one way that you can take a measurement of progress made. That is, if you are to implement a practice to improve a particular business value you will need to take a periodic reading to verify that the practice is working. This does not have to be quantitative. It may be qualitative in nature. Make it as simple as possible. For example, if you want to take a measurement to reduce cost, a simple (and rough) reading would be the number of hours put in for a major release.

We then must periodically measure our progress to decide if the practice we have adopted are helping us meet our goals.

It is worthwhile to note that none of these sources tell us exactly how to make the measurements and what those measurements should be. This is, of course, because it depends! It is also thought-provoking that Gilb and Brodie see measurements as missing from Agile, while many of us who are practicing see it as an integral part of the way we work. Is this similar to the outside world seeing Agile as 'cowboy-coding' while we know we are disciplined?