What metrics should be reported to management in an Agile software development environment?
On the Scrum Development mailing list, Charles Bradley asks:
A department has 3 dev teams that all report to one Senior Manager. That Senior Manager reports along with a few others to a VP.What metrics, if any, would you suggest to report from the team to the Sr. Mgr?
What metrics, if any, would you suggest to report from the Sr. Mgr up to the VP?
George Dinwiddie proposes a straightforward solution: ask management what metrics they would find valuable. In the case of this example, however, management doesn't know what kinds of metrics they want yet. They are wary of asking for metrics that break the principles and spirit of Scrum, and they are asking Bradley for advice.
Are there any kinds of metrics that should definitely not be reported to management in an Agile development environment? George Dinwiddie writes:
In general, don't report raw numbers. Those removed from the work often won't have the context to interpret them reasonably. And, upper managers don't have the time to do the analysis for themselves.
Specifically, according to Dinwiddie, team velocity is one number that should not be reported to management. Estimates about the future should be sent up rather than the velocity number itself.
Ron Jeffries writes that any metric whose intent is to compare productivity between teams is a bad idea:
Whatever you measure here, even if it were revenue dollars received cannot (I do not mean should not) be used to assess the teams' comparative "productivity". It will not work. It cannot work. I mean cannot in the sense that a rock cannot hang in the air unsupported. I mean cannot in the sense that a cat cannot build a bridge.A car salesman sells 30 cars a month. A realtor sells three houses a month. Which is more productive, and what should the other one do to become as productive as the one?
So what kinds of metrics should be sent up through the management chain? One approach, suggested by "karatasfamily", is to consider the questions that management will have about any software development project, Agile or not, then provide metrics that give insight into those questions. Some sample questions might be
- What are you doing?
- When are you doing?
- Are you on schedule?
- Are you on budget?
With this philosophy, the fact that a project is an Agile project is mostly encapsulated from management. The metrics reporting structure for Agile projects is designed to conform to management's current reporting formats, rather than management having to learn a whole new way of doing business. This attitude towards metrics is similar to the one described in Ken Schwaber's book Agile Project Management with Scrum.
The least-wasteful metric, though, is the one that never has to be created in the first place. In his book How to Measure Anything, Douglas W. Hubbard says,
If a measurement happens at all, it is because it must have some conceivable effect on decision and behavior. If we can't identify what decisions could be affected by a proposed measurement and how that measurement could change them, then the measurement simply has no value.
By this standard, every metric proposed—Agile or not—should be challenged with the question, "What is the decision this metric is supposed to support?" If no decisions would be affected by the metric, then the metric is useless and should not be reported.