Elizabeth Hendrickson On The Bugs Spread Disease
Elizabeth Hendrickson recently discussed the wasted hours she spent in bug triage meetings. In her blog testobsessed.com she addressed the problem that some organizations spend so much time and money on testing without really leveraging the information testing reveals.
As she explains in her article, there is a common and mistaken notion in the software engineering community that bugs are inevitable, and that not all of them can be fixed which is why "it makes sense to make a fix/defer decision on bugs based on some notion of ROI".
She has been employed in two companies where this line of thinking has turned out to be deathtrap. The companies have not been killed by the bugs directly. But, as Hendrickson explains, the bugs became a "pervasive infection" that slowed down productivity and paralysed the testers and engineers. In detail,
it was the hidden costs that drained us of life: hours lost arguing about bugs in bug triage meetings; hours lost stumbling over the same known issues again and again; hours lost fighting with a fragile and error-prone code base to make even small changes; hours lost cataloging and categorizing the backlog. It was demoralizing and immensely expensive.
Hendrickson draws a couple of conclusions from her experiences.
Cancel all the bug triage meetings; spend the time instead preventing and fixing defects. Test early and often to find the bugs earlier. Fix bugs as soon as they’re identified; attend to your broken windows early.
Readers of the blog have commented on this article. For example, Jim Gay adds:
I’ve seen bugs that indicated that the business process was wrong. For example an analyst tells you it should do X so you code X and then the users ask why it isn’t doing Y. Either a bug in the code or a bug in the process means something should be fixed.
Gabe Newcomb does not agree to all statements of Hendrickson:
This implies that all bugs are worth fixing — and that the effort to fix them is more important than any new functionality. That doesn’t match my experiences. The process of bug triage (as I am used to it) answers important questions such as when (and whether) to fix an issue, and where it fits in relation to other work. How would you propose answering these questions?
Steve Fenton is a developer who believes that all bugs should be fixed because:
The time it takes to fix a bug is nearly always less than the time it takes to allow it to endlessly loop around. As well as the customer impact, an old bug is discussed in meetings, is accidentally raised again and again by testers and then closed as a duplicate again and again by developers. The longer it hangs around, the more chance there is for it to cost more than it would have cost to fix it straight away.
nothing new under the sun
but being able to decide whether or not to fix a bug.
Alas, if there is such thing as a bug triage meeting (why not
have a meeting for while loops as well, or subclassing?), odds
are that people appointed to decide on bugs don't have much
grasp of their actual impediment on the development process,
and can't do better than "guess", as Hendrickson states in her
article: "The cost (not to fix a bug) is almost always far higher
than anyone might guess.".
A more relevant meeting would be for developers to explain
why such or such bug can't be quickly fixed.
This is a particular case of two problems:
- the general problem in software industry, that people who take
decisions usually don't have a clue,
- the eternal and universal problem of not having to choose
such or such clueless ideology, but of being able to have a clue.
This also reminds of an interview with Wayne Rosing, where he gives
a glimpse of how bugs were handled at Google after they got rid of
managers: "If something isn't right, even if it's in a product that
has already gone public, teams fix it without asking anyone."
Big plus one
Big plus one on that.
"And whatever you do, don’t accept the apologist’s excuse that bugs in production are inevitable, so the only pragmatic approach is to make a cost/benefit decision on whether or not to fix them. The cost is almost always far higher than anyone might guess."
On this point, I have a question. The problem she mentions repeatedly is the cost of the bug triage meetings. While I agree that it is feasible to eliminate bugs by using appropriate development methods, the reality is that many bugs have found their way into production and this will continue to be true for the foreseeable future. Only a small minority of development teams use appropriate techniques to avoid creating bugs or even to avoid letting them slip through the cracks into production. Yet, it isn't reasonable to expect businesses to spend a substantial amount of time fixing low-impact bugs. With that in mind, I think there is value in making a determination of which bugs are worth fixing.
Based on her own story, it seems the problem is that many organizations go about making that determination in a wasteful way. Then, they repeatedly report the same bugs and open the same questions about whether they are worth fixing. Those sound like procedural issues and not fundamental errors. That is not the same thing as determining the ROI for setting aside planned work in order to fix a bug. If the planned work would mean, for instance, we achieve fist mover advantage in some new market segment, then it may well be worth more to get the new feature into place before addressing the known bug. This has to be decided on a case-by-case basis and not in a blanket way, IMHO.
Brandon Holt, Preston Briggs, Luis Ceze, Mark Oskin May 21, 2015
Kai Kreuzer, Olaf Weinmann May 21, 2015