Measure Teams, Not Individuals

Posted by Mike Bria on Jan 31, 2008 05:50 AM

Community

Agile

Topics

Teamwork ,

Agile Techniques ,

Artifacts & Tools

Tags

Value & Metrics ,

Performance Evaluation ,

Management ,

TargetProcess ,

Planning

In response to requests from users of his company's agile project management and life-cycle product, Target Process, Michael Dubakov poses his warnings against the measurement of individual velocity or estimate accuracy on an agile project. His view is that measurement of these metrics will provide no more useful information than is already available with their team-level equivalents and may also encourage teams into behaviors that reduce effectiveness.

In a late 2007 post, Dubakov raised the issue of agile teams' desire to measure individual velocity. Using an example of two developers, Ted and Jerry, Dubakov illustrated his view that a set of historical 'individual velocity' measurements will provide a team no better mechanism to plan future iterations than does the team's overall velocity measurement:

If Ted completed several tasks with 40 hrs of effort in total and Jerry completed tasks with 25 hrs effort only, we should say that Ted has better velocity in last iteration. Does it mean that Ted is a better/faster developer? Not at all. There are hundreds of reasons why Jerry completed less ... OK, but what about average velocity? Surprisingly, Jerry’s average velocity is 54 hrs per iteration. Gosh! What happened with Jerry last two weeks? Will his average velocity help us to make correct iteration plan? If we sum up individual velocities all developers will it help us to create a better iteration plan? No, since we already have Iteration Velocity metric and it will be exactly the same.

More to his point, he continued by outlining two ways this metric is likely also to work against the ideal operational goals of an agile team:

1. a harmful focus on individual performance, over team performance, which he notes is likely to discourage team members taking time to help one another out
2. an increased tendency towards individual work assignment, over team commitment

Spurred by Michael's point and a related thread, James Carr soon added this reminder about the use of velocity in general:

Velocity is not about [assessing] performance… it’s about providing your customer with a close to accurate amount of “points” they can spend on features this iteration. Remember that.

In a more recent post Dubakov revisited the subject, this time adding a warning against the measurement of individual estimate accuracy. He first pointed out that this metric isn't even feasibly useful unless, first, the estimates were given by individuals and, second, the team tracks time spent for all tasks. The major problem with these conditions, many in the agile community would assert, is that both are likely to work against fundamental agile principles fostering teamwork and simplicity.

To illustrate the misleading effect which measurement of individual estimate accuracy may have, Dubakov goes back to the hypothetical developer Ted:

We can calculate all Ted’s assignments and spent time and define estimate accuracy for next iteration, let’s say it is 0.7.

OK, but how we can use this metric? Well, if Ted subscribed to 60 hrs work it means he will spend 85 hours and for 2 weeks iteration it means at least 5 hrs overtime. Ted should take this information into consideration and remove some tasks from his ToDo. This works if Ted estimate accuracy remains the same, but is that always the case? In reality Ted’s accuracy may vary from 0.5 to 0.9 and exactly for next iteration it may be 0.9 and in this case Ted will be able to do all committed work.

InfoQ's Deborah Hartmann takes Michael's point a step further, questioning the measurement of any time-based estimate accuracy at all, individual or team:

To calculate this kind of estimate accuracy, teams would have to expend effort to capture detailed "actual" hours, a practice not advocated by any Agile approach I've seen. The classic "units of work planned" vs "units of work fully completed" measures estimate accuracy in units of more value to the customer: work delivered (story points, ideal hours, bananas, whatever).

The tracking of estimate accuracy in terms of actual hours expended not only adds no more information to the team, it also imposes a new form of waste! I agree with Dubakov, Carr and others: I don't see how this could be a helpful metric for most teams, and I'm happy to see that it was quickly removed from TargetProcess once the point was raised. This is the kind of responsiveness to change that we expect Agile teams to exhibit.

Dubakov, Carr, and Hartmann all agree, with regard to both individual velocity and individual estimate accuracy, that measurement of these metrics not only provides no more useful information than is already available with their team-level equivalents, but may also have a tendency to encourage teams into behaviors that work against true agility.

1. Back to top

   Cultural Barriers are High

   Jan 31, 2008 1:56 PM by Mishkin Berteig

   Removing individual performance appraisals (and compensation - potentially) is difficult many cultures because of the substantial emphasis on individualism, heroics, competition, and self-fulfillment. Shifting to a true team-based system of commitment, performance, etc. is a process that can take a long time. For most organizations (of any decent size), changing things this fundamental (time tracking, etc.) can be a long arduous process! Often legal and HR have to get involved as well as management. Most organizations do not choose to tackle this early on. Only after Scrum or XP or Lean or any other agile method has been proven in terms of productivity, ROI, satisfaction levels, can these deeper changes be approached. That said, starting the conversations about measurement early is better than leaving them until they become a crisis! Here's a good quote about this sort of thing.

   Reply

2. Back to top

   Why not measure actual hours?

   Feb 5, 2008 8:57 AM by jared richardson

   The real reason we don't measure actual hours is the amount of time, and therefore dollars, it takes to capture actual time. Deborah says this above.

   To calculate this kind of estimate accuracy, teams would have to expend effort to capture detailed "actual" hours

   Our product line (6th Sense Analytics) is designed to measure the actual time invested by each developer without slowing the developer down. With a product like ours installed, the cost of measurement goes down to nearly zero. If you could get actual hours for free, would it be a meaningful metric? Would it be worth something to you to know in hours, instead of magic beans or bananas? I've found it's a very valuable piece of information. We do focus heavily on the team metric, not the individual. But it's much harder to build a meaningful team metric without measuring the individual. Here's our Use Case page for more information.

   Reply

3. Back to top

   Re: Why not measure actual hours?

   Feb 16, 2008 10:32 AM by Michael James

   The real reason we don't measure actual hours is the amount of time, and therefore dollars, it takes to capture actual time.

   Uh, no. The real reason is that bosses scrutinizing individuals is harmful to teamwork. I'm not inclined to mentor my teammates on their tasks if my boss is using some kind of analytical tool to measure my productivity on my own tasks. --mj

   Reply

• Older >