BT

Should we stop using Story Points and Velocity ?

by Anand Vishwanath on Dec 10, 2012 |

Agilists in the community are debating the use of story points and velocity, making people question the use of these in estimation and measurement of overall progress. The common underlying reason is these metrics are often prone to abuse and are seldom used effectively.

Joshua Kerievsky wrote a detailed account of his experiences in using story points and how they are prone to abuse. Here is an example he quotes where story points are prone to irrational inflation.

One day in 2004 Jim exhorted the team to go faster. This team had an average velocity around 52 points per iteration.Their velocity would fluctuate by a few points, but generally remained steady. Yet just weeks after Jim asked the team to "sprint", the team's velocity jumped up into the high 80s!

I asked someone what happened.

She looked at me funny and said "These days around here if you sneeze, you get a story point." I shook my head, amazed at how a mature agile team, a team that had been assessed, trained and coached by two excellent Industrial Logic coaches, and myself could so suddenly inflate their story point estimates to appear like they were going faster.

My confidence in story points and velocity calculations began to erode after that experience.  

 Joshua also highlighted how comparing teams by points is an anti pattern 

Over the years I've heard several managers at different companies ask, "Why is team X getting 24 story points done per sprint, while team Y is only getting 12 story points done? Those teams are roughly the same size, so why the discrepancy?"

Instead of treating velocity as a team-specific, capacity indicator, such managers fall into the trap of thinking of velocity as a performance measurement  

Uncle Bob Martin further confirms this in Joshua’s blog comments 

Many teams do struggle with story points. I have certainly seen teams devalue those points when their project managers try to get something for nothing by asking the team to go faster. I've also seen teams fake their velocity charts for the benefit of project managers who were more interested in form over function. For such teams, a different approach may be warranted. 

On a recent email thread on Scrum Alliance group Ron Jeffries was critical about Velocity as a metric.  

  • Velocity can be misused, and usually is;
  • Velocity can be gamed, and usually is;

And most of all, because it is working on the short end of the lever. What's important in Agile is steering the project by selecting things to do and things to defer, not working the cost side of the equation and trying to improve it.

A team that is focusing on velocity is not focusing on value. I wish I had never invented velocity, if in fact I did. 

He elaborates further in the thread saying

We find that most of the questions here are about how to improve estimates, so that they will be accurate. When we drill in, we see teams who are projecting when they'll be done and then pushing the teams to complete everything. We see them measuring teams on accuracy. 

We see product owners who are just doling out the backlog items with little or no control over anything, just "following the plan".

Scrum works best when the Product Owner uses the backlog creatively to deliver the best possible product. I have found that focus on estimates is not helpful to getting there.  

Mike Cohn recently posted on his blog an example conversation with one customer where he considered estimation to be important. The customer is quoted saying

First of all, in order for me to even consider giving future funding I need to know how close our estimating process got us this go-round.  When I ask the stakeholders for more money they’re going to think it’s going into a black hole if we only accomplish ½ of what we said we would.  I will manage this but I need to know we have a reasonable approach for coming up with high level estimates.  Second – to get follow on funding, I’m going to need to know roughly how much we need to collect (this is a long lead time activity).  We’ll have to improve our estimates if they’re nowhere near accurate and I’ll beat the pavement to get the funding.  That said, money doesn’t come for free.  I have to have some level of metrics that tell me throughout the project that we’re tracking to completion. In other words – I’ll look at the project burndown and watch our costs throughout the project.

Vasco Duarte suggests an alternative to story points .which is to estimate by counting stories .  

For the purposes of looking at the long term (remember: this only applies on the long term, i.e. more than 3 sprints into the future) estimation/progress of the project, you can assume that all stories are the same size, and can therefore measure progress by measuring the number of items completed per Sprint. 

Mike Cohn also commented on similar lines

I’d agree that Kanban teams do less explicit estimation. That’s often because they’ve made implicit estimates that all work is about the same size.

Neil Killick offers another approach to pricing  that does not need estimation. He contrasts an example to build a website for himself where the vendor builds the best possible solution given the revealed budget available and the Neil can chose to end the relationship if he is dissatisfied at any point in time.

He says that this option

Requires no estimation, design can change and emerge as we go along, embraces changes as I see the site evolve and shows a company wanting to work closely with me to achieve a result I am delighted with. One that is prepared to front extra risk (of losing money on the contract) because they are so confident in the quality of work they do and of the relationships they form with their customers.

Have you noticed practices such as velocity and story points estimation encouraging incorrect behaviour in teams ? What are your thoughts on some of the alternatives being suggested by the leaders in our community ?

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Story Points and Estimation are far from perfect by Mark Levison

Like Ron, Josh, Uncle Bob and others I've seen Story Points misused, but I've also seen them used well. When they're being misused i.e. perhaps as a measure of productivity or to push the team(s), I work to uncover the underlying problem - why are Story Points being used in this way? Is the problem soluble - I abandon points only when we can't cure the abuse.

Cheers
Mark

Problem is management. Replace velocity. by Mike McGranahan

Their is only one problem here and that is management's misunderstanding of the intention of velocity, due to a larger misunderstanding of the goal of Agile. Management believes velocity is an absolute measure that represents productivity, how much value the team is generating for a period of time. But actually, velocity is merely an intermediate value relative to the team's estimates useful in determining the team's estimation accuracy.

Velocity is a tool to buttress the regular, predictable delivery of features to end users to close the feedback loop. The scarier problem with a management that abuses velocity is that this is a sign of a deeper misalignment: that feature development trumps a predictable feedback cycle.

My solution would be to replace velocity with something less semantically confusing. I'd rather relegate velocity to the status of an "intermediate variable" to generate a Planning Factor (or Estimation Accuracy Factor), which is simply the scheduled points divided by the completed points. Above 1.0 means you estimated that you could complete too much; below 1.0 means you estimated too little. That's the high-level number you can talk about when management is around. When it comes to iteration planning, you inevitably refer to the calculated velocity to determine by how much to adjust your capacity. Even then, with an ignorant management I'd refer to it more ambiguously as Reference Capacity.

Trend is important by tushar jain

To compare teams(if I have to do it), story points or velocity is not the criteria I use in my projects but trend in velocity. I feel trend is more important than absolute velocity (architecture-soa-bpm-eai.blogspot.com/search?q=...)

Velocity used alone acts as a regulator by Guy Winterbotham

Goodhart's law arose in economics. It states:

`As soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.'

Put another way by Professor Marilyn Strathern FBA:

`When a measure becomes a target, it ceases to be a good measure.'

You need to have business value as the measure and velocity as an indicator. Since velocity is the handy metric I feel some teams start substituting velocity for business value.

You get what you measure by Dean Schulze

They must have forgotten that they will get what they measure. Measuring story points is no different than counting lines of code.

Re: Problem is management. Replace velocity. by Piyush Kumar

The problem lies in the interpretation of velocity as a measure of progress. In addition, if there is force applied to increase the velocity (without any other factors changed significantly to support this), it leads to abuse. It would be the same case with other measures such as counting number of stories as well. As long as the measures are not left as just measures instead of trying to use them as targets, they will be abused. You can bring any measure into picture and it would get abused unless the management understands its objectives.

You can introduce a new measure instead of velocity. It would work fine in the short term but as the usage will increase, there will be similar problem patterns that might repeat. As more people adopt a new thing, the underlying principles and objectives are not understood well by everyone. This is at the root of the problem.

Re: You get what you measure by Dirk Detering

Right ... If I measure the velocity in my car as being too slow, I have to press the pedal to lead more gasoline into the combustion engine. In the end I can see if it works by looking at the velocity again. (If it does not work, something may be broken).

The car doesn't do anything by telling it "go faster".

(BTW: This reminds me to the term "up to eleven" ... dunno? Google for it)

Understand agile development by Patrick Kranz

Like it was already said, the problem is not the estimation but the lack of understanding in the managemeht what it means. Personally, I value estimations to be the very best tool to see that everybody in a development team is on the same page and has the same understanding of a user story and what needs to be done.

The problems described here in the article indicate that people try to apply waterfall patterns to agile development. If they take the velocity, combine it with story points and translate this to days a feature will take to develop, just in order to see the overall progress, then the problem is in the organization and not in the process the developers are using.

Those people need to understand how agile development works because they simply don't get it yet. They are still thinking in terms of shipping a product when the TV spot is booked and not when the product is actually finished. And as long as they do not understand the difference between agile and waterfall, they will use a different metric if there is no velocity anymore.

Personally, I would not abandon estimations if they help me and my team in understanding the stories. I think a really good scrummaster could help here. At least it helped on the projects I've been on.

Greets,
Patrick

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

8 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT