Estimating on agile projects: what’s the story, what’s the point?
Before you commission a painter to decorate your home or a mechanic to fix your car, you get an estimate from them, right? You need to know how much it’s likely to cost and how long it might take. It’s just common sense.
What does experience tell us, however? How close are those original estimates to the final bill? It’s all too likely that the painter will find loose plaster that needs removing and the wall needs rendering and re-plastering. The mechanic is sure to find additional work required to get your car roadworthy again. In a 1951 cartoon for the New Yorker magazine, Syd Hoff drew a mechanic saying to his customer,
“Of course, that’s only an estimate; the actual cost will be more”.
If the painter or mechanic tells us in enough time, we can choose not to take on the extra work … and yet, all too often, we feel we have to fix these additional things. Who wants to live in a house with a potentially damp wall or drive in a car with potentially faulty steering?
How do we overcome this? A common reaction is to insist on a full and final fixed-price estimate, or ‘quote’ as it’s commonly called; so they work harder and longer to get more accurate estimates. Yet however hard they work, they still cannot really predict the unexpected.
In the life of projects, this is just the same
Traditionally, project managers tend to focus on creating detailed estimates that can withstand scrutiny from the finance team. Of course, this is based on ‘known knowns’ with some contingency for the ‘known unknowns’ … and as Donald Rumsfeld is famously quoted “there are also unknown unknowns – there are things we do not know we don't know”. Like the tradespeople above, we can never really predict the unexpected.
The more we invest in creating elaborate estimates, however, the more problems we cause. Detailed estimates can be seen as a binding quote, a target that distracts from delivering value, and focusing on delivering something—or even anything—to the agreed date and cost.
We beat ourselves up in post-implementation reviews, and tell ourselves to just try harder; Einstein, however, defined insanity as “doing the same thing over and over and expecting different results”; so there must be a better way.
Surely this isn’t true for agile projects? Don’t we just start with a high-level scope and work it out as we go along? Well, yes and no (or as they would say here in New Zealand, “yeah … nah”). While we don’t hamstring ourselves by creating detailed estimates, it is still vital that we get a feel for the size of the work we’re considering, and here’s why …
Why we need estimates
Forget about estimates being used to build beautifully crafted Gantt charts that force us to focus on tasks and not outcomes. There are three outcomes that need us to get a rough handle on how big something is.
- When we’re considering the justification for a proposed project, we need to understand the likely costs in advance, so that we can decide whether it is worth the investment.
- When we’re launching new or improved products to market, we need some idea of roughly when significant features might be ready for release, so we can plan associated activities.
- When we’re prioritising work, the product owner needs to understand the cost, and the team needs to understand the value, of each story (or backlog item).
It’s also interesting to note that estimating can be a really healthy activity, so long as the whole team collaborates on it together. It helps foster a buy-in from the whole team and ensure that everyone gets a common understanding of the scope and value to be delivered.
However the label of ‘estimates’ can be distracting.
Use sizes rather than estimates
To avoid setting an expectation that we’re talking about cost and time, when we estimate the complexity of a story, some of us like to refer to this as ‘sizing’ rather than estimating. In the 90’s when I first used Scrum and XP—before they were even called agile practices—we sized stories using t-shirt sizes (S, M, L, and XL).
Now, however, we use story points – a way of sizing stories relative to each other – so we find what will be the simplest story, size that as 1-point story, then compare another story to that, if it’s more complex than maybe it’s a 3-point story.
To make things more interesting, we don’t use a simple sequence like 1, 2, 3, 4, 5, etc. – instead we use a modified form of the Fibonacci sequence like 1, 2, 3, 5, 8, 13, etc. (as seen in ‘The Da Vinci Code’). This makes the jump between numbers larger the higher we go and drives us to choose which is smaller or larger.
While this is not an exact science, it is more than good enough for the three outcomes mentioned above, and as John Maynard Keynes said, “it is better to be roughly right, than precisely wrong”. This means, though, we still need to translate story points to rough time and rough cost.
Another practice central to agile projects is establishing the ‘definition of done’, that is to have a comprehensive understanding of everything required to say that a story is done and releasable, including items like user documentation, translation, advertising, etc. etc.
Provided we have a good definition of done, we can then take a couple of sample stories and calculate the effort required. From that we can get rough cost estimates for an investment decision, rough timing for release planning, and enough understanding to assist in story prioritisation.
Some, however, still find estimating by story points a distraction, and one response has been the debate around #NoEstimates.
So what’s with the #NoEstimates debate
If you have been following any of the major influencers on Twitter, you might have noticed some getting involved in discussions with the hashtag of #NoEstimates. While this sounds like a call to cease estimating altogether, it is really a call to stop sizing stories and instead just start developing.
This discussion arose, and has gained momentum, because the experience of those working on large projects has been that whether you size by story points or just count the number of stories, the tracking of throughput is about the same.
While this might partly be down to the more experienced delivery teams decomposing stories to consistently smaller sizes on a just-in-time basis—as this enables greater throughput of potentially shippable increments—the reporting looks similar even on projects still working with a range of very small to moderately large stories in each iteration.
While this makes story prioritisation simpler and enables the delivery team to get on with it quicker; our poor product owners and sponsors still need to know roughly how much it will cost and how long it will take. The good news is that we can still do this based on counting the number of stories, once we have sufficient metrics to make this feasible.
For any new team starting out on your agile journey, however, I would still strongly recommend you size with story points until you’ve reached the experience and maturity levels of these teams.
So to sum up, and to mangle a famous quote from General Dwight D. Eisenhower: "In preparing for [projects] I have always found that [estimates] are useless, but [estimating] is indispensable"; we still need to do it, whatever we call it and whatever we’re counting when we do it.
I am indebted to Esther Derby (estimates become targets), Mike Cohn (estimates for release planning and prioritisation), Ahmed Sidky (whole-team estimation), Martin Fowler (story points), Ian Mitchell (definition of done), Vasco Duarte (story count vs story points), Stephen Forte (metrics vs estimates), Neil Killick (experienced teams define smaller stories), amongst many others, for sharing their thinking on this topic – and I have linked to their respective articles in context above.
About the Author
David Morris is an independent business agility practitioner, coach, and instructor; working through his company Sophorum, delivering business analysis, team leading, coaching, and training services to customers throughout New Zealand. With nearly 30 years’ experience in project delivery, he has worked in and led teams and run his own business across strategic, business, and technical projects following structured, iterative, and agile methodologies.
David is a qualified CBAP, ICAgile Certified Practitioner, Certified ScrumMaster, and an IT Certified Professional. He runs study groups, professional development workshops, and bootcamps; helps organise events for Agile Auckland and IIBA NZ; has spoken at conferences and events in Europe and Australasia; contributed to several books (including the ‘Agile Extension to the BA Body of Knowledge’); and is an active blogger, tweeter, and Wikipedia editor.
Apply the 5 whys
So the key point in the article is:
"WHY WE NEED ESTIMATES
1. When we’re considering the justification for a proposed project, we need to understand the likely costs in advance, so that we can decide whether it is worth the investment.
2. When we’re launching new or improved products to market, we need some idea of roughly when significant features might be ready for release, so we can plan associated activities.
3. When we’re prioritising work, the product owner needs to understand the cost, and the team needs to understand the value, of each story (or backlog item)."
I dispute that all three needs are best addressed by estimates:
1. In practice, any software development project that has so narrow a profit margin is doomed to failure. Software development cost is unpredictable, not just because developers do not know what they do not know, but also because when given working software customers will always be discovering that they really did not need exactly what they asked for but they do need something they did not know to ask for until they tried using the software. Also, people, markets, competitor, etc. change.
A software development project is an investment. If the ROI does not potentially dwarf the cost, it is best to deploy your investment in something that will. Furthermore, every month into the project, you should have a much better idea of whether things will work out than you did the month before. Just start, deliver working software quickly, obtain and respond to feedback quickly, and pull the plug quickly if necessary. Start, inspect and adapt is much more realistic and much less expensive than wasting time and money building unreliable estimation models (which, by the way, are even more inaccurate on the return side than the cost side anyway).
2. Again, by getting started immediately and delivering working software, within a month or two, you will have a much better idea of when any particular feature will be delivered than playing the estimation game. The fact is that you will almost always have the feature ready and working by the release date - it just might not have all the bells and whistles.
3. Doing the first vertical slice of a feature will tell you much more about the cost of getting the feature totally working than story pointing will. And you will be 2 weeks closer to delivering it, to boot. There is nothing wrong with implementing the first vertical slice of several features in an iteration, and then making an informed decision about which one(s) to complete first.
The real reasons that businesses want cost estimations are:
A. Antiquated, big-bang budgeting practices.
B. CYA - so the business can always blame development if things do not work out.
Re: Apply the 5 whys
The "need" for estimates
ROI = (Value - Cost)/Cost
As for the issue of accounting needing an accurate estimate for a project. This statement confuses Budget with Funding. Accounting needs to know the "funding" for the project. The management of the project is done in units of "budgets." Budgets overrun all the time. More funding is applied. The management of projects with budget provides feedback from comparing actual to budget for all the elements of the project. Mike Clayton has a nice "periodic table" for the elements of a project mikeclayton.files.wordpress.com/2013/11/pm-peri...
Mixing financial accounting and project budgets is common in some domains. Don't let that happen. Flow down the budget, let finance handle the money. This is the basis of all government and commercial contracting budget <> funding.
Finally the Rumsfeld quote is simply bad management on his part and the part of those using it. If Rumsfeld had read history, literally "The Histories" en.wikipedia.org/wiki/Histories_(Herodotus) he would have read the advice of Alexandr the Great - "Don't Go There."
There are two kinds of uncertainty in the project domain (actually in the mathematics of uncertainty) - reducible and irreducible. Both create risk to the project. Reducible is addressed by spending money or setting aside Management Reserve. Irreducible is address by "margin," either schedule, cost, or technical.
In order to have a credible plan to handle anything on a project, including the probabilistic estimates needed for proper governance, there are three approaches
1. We don't know upfront - we didn't sufficiently plan
2. We don't know during execute - the uncertainty emerged
3. We don't want to know - knowing will cause the project to be canceled before it starts.
The notion of "not wanting" to estimate is number three, since the "need" for estimates is fundamental at any business process - we can't know the value of anything until we know what it costs. This basic notion of business seems to be intentionally ignored in the No Estimates community.
I like story points
We told the stakeholders we needed 3 weeks before we could give them an "end date" - and we spent 3 weeks building, until we got to a point where we could measure our velocity (in points per week). Then I made projections - optimistic was velocity + 20% and pessimistic was velocity - 20%, giving a date range of when we would be "done" (knowing full well of course that things would change over the coming weeks as we got to understanding the scope better).
The amount of effort expended was minimal, and it was just about scientific enough to stop the stakeholders worrying (and basically keep them off our backs so we could get on with it!). For me it's a nice balance because it is being honest about uncertainty (no fixed date, just a range) - but it's way more palatable to a stakeholder than "it will be finished when it's finished".
Re: Apply the 5 whys
1) Your points seem predicated on a perception that sizing should be considered as "wasting time and money building unrealistic estimation models". It does not take long to ask the team how complex they think a piece of functionality is, flashing some planning poker cards (or fingers, whatever), agreeing a size or discussing what's involved if the variance is large. The team also benefits from gaining a better understanding of work coming up, so their horizon is not too close. This is beneficial and doesn't stop the team from getting started, so I see this as a win-win.
2) The issues does not have to be so much on whether the ROI on an individual project is worthwhile, and is more usually because organisations are assessing a portfolio of suggested product and business changes, and not many organisations have the budget and people to do everything they want all at once. They need a way of comparing unlike projects, and one way of doing that is to have a rough feel for the costs and timescales, as well as the likely benefits.
Re: Apply the 5 whys
Olav Maassen, Liz Keogh & Chris Matts Mar 08, 2014