Is Estimating A Wasteful Practice?

For multiple reasons I'm starting to lean toward T-shirt sizing for product backlog estimates. One result: people seem to feel more free to throw an estimate into the ring (with planning poker, for example) when it's not a number. At this level of planning we know the "accuracy" is illusory, but it seems hard to avoid when estimating with numbers.

PS: Tool makers: please give us a way to translate t-shirt sizes into numbers for release forecasting!

We have used equal sized stories in two projects very successfully. Out story concept was influenced by the feature concept from Feature Driven Development and we established tha rule that such a story should be implementable in one day.

Now now, Deb. :)

Seriously, though, I'm liking Amit's approach to regularizing the size of items in the backlog (to the extent possible), as it makes forecasting (which is what most people want from estimates) a much simpler process. And it's substantially in line with the queueing theory notion that the more regular the units in a queue, and the smaller those units are, relative to the width of the queue, the less slack you need for optimal throughput. (ie, the more efficient the queue is.) In my experience it works for plumbing, highways, but also software teams.

So, of course it isn't really estimation-less.



It is in the same spirit as pair-programming - this practices eliminates the need for code-reviews. In a rather zen-like manner, it does this by doing more of it - a pair of developers constantly review code, all the time.



Breaking stories down into small chunks is a Good Thing (queueing theory). If nearly all stories can be broken down to this level, then many other benefits follow (as mentioned on the various blog posts above). And there's the gotcha - to achieve that homogeneity - one has to constantly estimate.



It is just that the focus is different - improving throughput, as opposed to meaningless prediction - and this is a simple way of getting there.

It seems some are "not estimating" and using variously sized stories. This does seem to remove estimation effort. Though how outside stakeholders plan for release isn't obvious.

Others do it with "same sized" stories. Who gets to estimate that? How long does that conversation take, to get stories to uniform sizes? I'm not sure this variant eliminates estimation.

I'm very much on the "no estimates" side, in fact I recently wrote a post about what I call the "estimation fallacy" which you can read here: blog.robbowley.net/2008/6/6/estimation. In short I argue that not only is it a waste of time (as software is too unpredictable), it is also dangerous as people make commitments (whether you like it or not) based on your estimates.



Just a few days ago I posted a response to Jeff Atwood(Coding Horror)'s post arguing developers need to write exhaustive lists of everything they need to do and that is the only way to get reliable estimates. My argument is the same as above.
Jeff's post
www.codinghorror.com/blog/archives/001161.html
My response
blog.robbowley.net/2008/8/11/lists




Its a bit of an obsession of mine at the moment, I have a few more posts up my sleave around the issue and would be very grateful for feedback you have on my thoughts.

The frustrating fact about not estimating to avoid people making commitments based on those estimates, is that in most economic scenarios, dollars are allocated based on commitments, and it's hard to get dollars later if the belief turns out to be false. So commitments are made based on confidence in the estimates. Even if we know they're false, the whole business world is rife with a contract-oriented mindset that essentially is promises of fees for a service or product.



So how do we provide that customer with confidence that they're going to get value for their money... and how much value for how much money, so they can know how much to allocate? These are all questions that, even if estimation is total crapolla, we still have to answer this basic problem in another way. Just "not estimating" and assuming it'll all work out only works inside of a very high-trust context, and how do we handle questions of allocation during the trust-building phase with a customer?

So, you're going to remodel your old house. It's a two-story Victorian, and you want to knock out half of the back exterior wall, expand the kitchen, and a 3/4 bathroom, and put a new bedroom above all that on the second floor.


You call up a contractor, and she says that she won't give you any estimates, but she will show clear progress on a weekly basis, and you're free to kill the contract at the end of any given week if you're not satisfied.


So you begin work. They blast out the back wall, frame up the new rooms, and they're beginning work on the kitchen expansion, when you begin to realize that you're burning through your savings faster than you expected, and you're not sure if you'll be able to complete the job as you'd like. In fact, you're beginning to worry if you'll be able to get the thing completed in any useful way at all. At this point, you ask the contractor for an estimate of the work remaining to be done, and she gleefully responds, "I don't give estimates, but you can cancel at any time!"


So what do you do? (And if your response is that this is nothing like a software project, then I would tell you that you've either not had much experience with remodeling projects, or software projects, or both.)

I wrote about this here.



Rather than making commitments through estimation and prediction, the same commitment and reliability can be achieved through measuring performance of throughput and cycle time of features, and using that information to make forecasts.



This technique requires either using the technique descried by Amit to same-size work items, or by classifying work items into similar sized categories (e.g. T-shirt sizing)

"You call up a contractor, and she says that she won't give you any estimates, but she will show clear progress on a weekly basis"

Aside from the whole "software development is like building a house" fallacy - the kind of software projects being discussed here don't just show progress, they deliver working software, and quickly. If your project requires several iterations (however long they may be) before it delivers any kind of value, it's usually time to take a critical look at the practises being used.



I see your home improvement plan as being equivalent to spending X amount of time on requirements gathering, followed by X amount of time in development, followed by X amount of time testing, etc. Of course if you cancel such a project halfway through you end up with bugger all.



Jim

Wait a sec, Kurt.

Run your scenario again, but this time have the contractor give you a super precise estimate up front.

What's the difference? Aren't you still screwed? Wasn't the error allowing the contractor to have too much work in process such that you weren't actually able to take advantage of the "cancel at any time" clause?

Honest questions.

But at least in that case of an upfront estimate you have information (flawed and inaccurate though it may be)upon which to make a decisions (secure additional funding, re-scope, or simply kill the whole thing). Granted if the contractor's cost estimates are too low, and the contract is time and material, then you could still end up being screwed. Or you put the risk on the contractor and try fixed price. Either way estimation (regardless of which approach) is simply a way for parties to understand risk. The fallacy is the idea that estimates have some inherit accuracy that allows you actually to manage risk. That is where practices and processes come in.

Additionally, the approach(es) discussed, at least the overall ideas, don't *necessarily* eliminate <em>forecasting</em> from the equation.



They are more about changing the way we as go about doing our forecasting.



I was working with a company a few months ago operating on a 2 month release cycle. As you'd expect, they wanted to have at least some idea of what was coming out of the other end of the 2 months. We successfully used a "numbers"-less estimation approach to achieve that; "successfully" in that we were able to give a good enough idea, then go about using "pull-based" iterations to continually grow a product over the 2 months that everyone was happy with. Of course, the end result was different than the original projection, but it was better.



Either way, the key still is focusing on *really* delivering running/tested features (WORKING SOFTWARE) each step of the way. If you release as you go, every iteration, great. If you have more long-term release cycles (assuming they're no longer than a few months), great too.



The point is to be less mathematical about how you go about deciding what you sign up for.

The question is not whether micro-iterations and estimation-less development would work. The question is where it would work and what type of social and business barriers stand in the way.

Funding is one of the main indicators of whether a software environment would be able to adopt this model. If your funding machine can be recalibrated to work in a negotiable scope or or time & matieral fashion, then there is no reason that "micro-iterations" shouldn't work. But convincing people to pay for software in this fashion is not always easy.

(Also posted on the XP list)



Seeing as all the Industrial Logic folks are experts at (I)XP, wouldn't this be considered an expert-level practice?



If a team is in the Shu or Ha stages of learning, I'm not convinced that they're good enough at breaking down the work into small enough stories to use a pure pull process. In IL's case, Josh was the Customer, n'est-çe pas? How many teams have someone that advanced in that position?



Dave Rooney

Mayford Technologies

I suspect that in the examples cited above there are implicit assumptions being made by the product owners. Imagine I have 2 equally important (from a user's point of view) stories in my backlog. As a product owner I want to deliver value early and often, so without an estimate of cost (and presumably there is no estimate for risk either), how would I be able to determine which story to work on first? I know I'd be more comfortable with the "best guess" (this is what estimates really are) coming from the people who will be doing the work...

Gavin, very good point. As with all thing agile, we need to be pragmatic. The root of all of this is: For the most part (ie, our 'usual process'), we don't get muddied in 'hard estimates'. Pragmatism injected then...of course, when a question such as "we can only do this or that, which one can we get quicker' is asked, then the team answers.



You're statement highlights a key concept: the idea of "Bargain Hunting" (as Josh K likes to say). Basically, at the heart of successful agile is a die-hard mentality amidst the team to constantly ask and answer the question "how can we deliver the most value in the shortest time". At its best, what this looks like is a team that's constantly trimming the "Feature Fat" off each and every story (another Josh K term) so that it can get the important stuff quick, or at least first.



This may sound odd and counter-intuitive, but I've had experiences where putting numbers (estimates) on the cards actually acted as an inhibitor to this "bargain hunting" culture; rather than looking at a story and always building the "simplest thing that could work" (ie, as "fat-free" as possible), they'd just build a "5", or a "3" or "8". One of the downsides to the number - the "crutch-factor".



All, I'd like to thank everyone for the fabulous discussion around this topic - advancing our craft lies in these debates, please keep it up! Also, remember, the topic is also running on the XP Yahoo Group, jump in there too.

Here here Bruce!!

I think this is the best answer I've heard to the argument presented by Kurt.

I guess at least with the super precise estimate up-front the home-owner can later sue the contractor, spend a lot of money on lawyers and time in court (more wasted productivity) and then eventually settle and therefore only get a portion of their investment back and be jaded with any future contractors.

The analogy made would work better if I was going to build this addition, I might start with the kitchen remodel, but I would keep in mind that I would have a second floor going in and consider having some underlying architectural (evolving architecture) support in my kitchen, but I wouldn't rip everything down and have a gapping hole. Of course I may not put the pipes in for the bathroom that run through the kitchen, so I may have to go back and refactor that, but it's a trade-off.

Keep in mind Kurt that the goal is to have something "potentially shippable" at the end of each week not something that has to wait until the finished product (Jim makes this point too). The idea is that I can walk away at that point because I do have value and something I can use. For example, I added the new island and sink in the kitchen. Next week I may prioritize the removing the old sink and counters and replacing them with the new counters and cabinetry. I think "gut feel" the contractor could tell me if it's doable (commitment) and then give me some warning if not (manage client expectations). I can also see progress during the week and give my input if needed.

Regards.

Hi Bruce! I always love your questions and comments...



Anyway, to your first point, I might be screwed in that I wouldn't have my remodeling work done, but at least I wouldn't be out of money with a house half torn apart.



Your second point resonates more strongly with me - I would indeed be in better shape if I could work more closely with the contractor, so that I didn't find myself in a bad position. But, to pay for these contractors, presumably I have to go to work every day. So, I can't check in with the contractors all the time; I have to have some notion of an iteration, and I'd like some idea of a plan with estimates when I begin that iteration - even if the iteration is just a day.



The analogy goes further. Once the contractor begins framing the adition to the kitchen, I'm not left with something useful if I need to fire them at the end of the week - I'll either need to find someone else to finish the job, or else I'll have to tear it all down and rebuild my back wall - assuming I haven't run out of money. Likewise, there are *some* classes of software for which we can meaningfully talk about creating potentially shippable software in short iterations. But there are lots and lots of projects for which this simply isn't true.



As an example, I'm currently working with a software company to help them transition to more agile processes. They have a big huge product, with a big huge complicated codebase. Now, when the sales people go talk to customers, they need to be able talk in terms of the 5 or 10 major features which will be coming out in the next product release - they can't meaningfully talk about a 5000-item prioritized backlog. For each feature, we could argue about whether or not some particular little thing gets added or not, but at some point, if I've lost, say, 25% of the backlog items associated with that feature, then from the perspective of a salesperson or a customer, I can no longer realistically say that that feature exists.



I guess it comes down to the reality that in many environments, there are people who need to make commitments based on our work, in order to generate revenue to keep paying for our work. And this requires plans, which require estimates. The estimates certainly don't have to be perfect, but from a business perspective I need *something*.

Software is not a house. Sorry Kurt, but its such a crap analogy. I use "building a house" to explain what making software is exactly not.

If software was a house and I was a contractor given the above conditions I would build something that was usable as soon as possible (e.g. a wooden shed extension), see what the customer thinks and then rebuild it every two weeks adding more and more of the requirements and responding to the feedback as we go. In the end we'd have as much of the requirements that were possible with the allocated money and time frame. We'd have not gone over budget, been able to respond to unexpected events (e.g. finding out the foundations are not sufficient) and the customer could change his mind (on windows, room layouts etc) as we go.

Software is not like building a house - we have the luxury that it's relatively inexpensive (if we've followed good design principles), to go back and change it as often as we like. I say, tell me honestly how much you've got to spend/how much time you have and I'll give you the best possible value for those conditions. If an unexpected event occurred which means we can't deliver it well, it would have happened if we'd estimated it anyway. At least my way we're more likely to find out sooner and hopefully save the customer some money/embarrassment.

And, by the way, I have lots of experience working on large applications and have seen both approaches attempted. I'd say that if you think the software house analogy sticks then it says a lot more about your approach or the people you've worked with. Building software in an iterative (not incremental) style with regular integration and customer feedback works and I know this because I've been doing it with great success for the past year and a half. Our customer's have never been happier...

You don't really need a formal estimate that is a 'guaranteed' price, but ...
You and your contractor should have figured out the cost of materials (COTS and production hardware in the software world?) and the contractor should have been able to give a 'ballpark (gut) estimate', similar to what Josh Kerievsky was talking about in the article.