Should we stop using Story Points and Velocity ?
Agilists in the community are debating the use of story points and velocity, making people question the use of these in estimation and measurement of overall progress. The common underlying reason is these metrics are often prone to abuse and are seldom used effectively.
Joshua Kerievsky wrote a detailed account of his experiences in using story points and how they are prone to abuse. Here is an example he quotes where story points are prone to irrational inflation.
One day in 2004 Jim exhorted the team to go faster. This team had an average velocity around 52 points per iteration.Their velocity would fluctuate by a few points, but generally remained steady. Yet just weeks after Jim asked the team to "sprint", the team's velocity jumped up into the high 80s!
I asked someone what happened.
She looked at me funny and said "These days around here if you sneeze, you get a story point." I shook my head, amazed at how a mature agile team, a team that had been assessed, trained and coached by two excellent Industrial Logic coaches, and myself could so suddenly inflate their story point estimates to appear like they were going faster.
My confidence in story points and velocity calculations began to erode after that experience.
Joshua also highlighted how comparing teams by points is an anti pattern
Over the years I've heard several managers at different companies ask, "Why is team X getting 24 story points done per sprint, while team Y is only getting 12 story points done? Those teams are roughly the same size, so why the discrepancy?"
Instead of treating velocity as a team-specific, capacity indicator, such managers fall into the trap of thinking of velocity as a performance measurement
Uncle Bob Martin further confirms this in Joshua’s blog comments
Many teams do struggle with story points. I have certainly seen teams devalue those points when their project managers try to get something for nothing by asking the team to go faster. I've also seen teams fake their velocity charts for the benefit of project managers who were more interested in form over function. For such teams, a different approach may be warranted.
On a recent email thread on Scrum Alliance group Ron Jeffries was critical about Velocity as a metric.
- Velocity can be misused, and usually is;
- Velocity can be gamed, and usually is;
And most of all, because it is working on the short end of the lever. What's important in Agile is steering the project by selecting things to do and things to defer, not working the cost side of the equation and trying to improve it.
A team that is focusing on velocity is not focusing on value. I wish I had never invented velocity, if in fact I did.
He elaborates further in the thread saying
We find that most of the questions here are about how to improve estimates, so that they will be accurate. When we drill in, we see teams who are projecting when they'll be done and then pushing the teams to complete everything. We see them measuring teams on accuracy.
We see product owners who are just doling out the backlog items with little or no control over anything, just "following the plan".
Scrum works best when the Product Owner uses the backlog creatively to deliver the best possible product. I have found that focus on estimates is not helpful to getting there.
Mike Cohn recently posted on his blog an example conversation with one customer where he considered estimation to be important. The customer is quoted saying
First of all, in order for me to even consider giving future funding I need to know how close our estimating process got us this go-round. When I ask the stakeholders for more money they’re going to think it’s going into a black hole if we only accomplish ½ of what we said we would. I will manage this but I need to know we have a reasonable approach for coming up with high level estimates. Second – to get follow on funding, I’m going to need to know roughly how much we need to collect (this is a long lead time activity). We’ll have to improve our estimates if they’re nowhere near accurate and I’ll beat the pavement to get the funding. That said, money doesn’t come for free. I have to have some level of metrics that tell me throughout the project that we’re tracking to completion. In other words – I’ll look at the project burndown and watch our costs throughout the project.
Vasco Duarte suggests an alternative to story points .which is to estimate by counting stories .
For the purposes of looking at the long term (remember: this only applies on the long term, i.e. more than 3 sprints into the future) estimation/progress of the project, you can assume that all stories are the same size, and can therefore measure progress by measuring the number of items completed per Sprint.
Mike Cohn also commented on similar lines
I’d agree that Kanban teams do less explicit estimation. That’s often because they’ve made implicit estimates that all work is about the same size.
Neil Killick offers another approach to pricing that does not need estimation. He contrasts an example to build a website for himself where the vendor builds the best possible solution given the revealed budget available and the Neil can chose to end the relationship if he is dissatisfied at any point in time.
He says that this option
Requires no estimation, design can change and emerge as we go along, embraces changes as I see the site evolve and shows a company wanting to work closely with me to achieve a result I am delighted with. One that is prepared to front extra risk (of losing money on the contract) because they are so confident in the quality of work they do and of the relationships they form with their customers.
Have you noticed practices such as velocity and story points estimation encouraging incorrect behaviour in teams ? What are your thoughts on some of the alternatives being suggested by the leaders in our community ?
Story Points and Estimation are far from perfect
Problem is management. Replace velocity.
Velocity is a tool to buttress the regular, predictable delivery of features to end users to close the feedback loop. The scarier problem with a management that abuses velocity is that this is a sign of a deeper misalignment: that feature development trumps a predictable feedback cycle.
My solution would be to replace velocity with something less semantically confusing. I'd rather relegate velocity to the status of an "intermediate variable" to generate a Planning Factor (or Estimation Accuracy Factor), which is simply the scheduled points divided by the completed points. Above 1.0 means you estimated that you could complete too much; below 1.0 means you estimated too little. That's the high-level number you can talk about when management is around. When it comes to iteration planning, you inevitably refer to the calculated velocity to determine by how much to adjust your capacity. Even then, with an ignorant management I'd refer to it more ambiguously as Reference Capacity.
Trend is important
Velocity used alone acts as a regulator
`As soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.'
Put another way by Professor Marilyn Strathern FBA:
`When a measure becomes a target, it ceases to be a good measure.'
You need to have business value as the measure and velocity as an indicator. Since velocity is the handy metric I feel some teams start substituting velocity for business value.
You get what you measure
Re: Problem is management. Replace velocity.
You can introduce a new measure instead of velocity. It would work fine in the short term but as the usage will increase, there will be similar problem patterns that might repeat. As more people adopt a new thing, the underlying principles and objectives are not understood well by everyone. This is at the root of the problem.
Re: You get what you measure
The car doesn't do anything by telling it "go faster".
(BTW: This reminds me to the term "up to eleven" ... dunno? Google for it)
Understand agile development
The problems described here in the article indicate that people try to apply waterfall patterns to agile development. If they take the velocity, combine it with story points and translate this to days a feature will take to develop, just in order to see the overall progress, then the problem is in the organization and not in the process the developers are using.
Those people need to understand how agile development works because they simply don't get it yet. They are still thinking in terms of shipping a product when the TV spot is booked and not when the product is actually finished. And as long as they do not understand the difference between agile and waterfall, they will use a different metric if there is no velocity anymore.
Personally, I would not abandon estimations if they help me and my team in understanding the stories. I think a really good scrummaster could help here. At least it helped on the projects I've been on.
Roy Rapoport Aug 28, 2014