Empirical Studies on Software Quality Mythology
- Share
-
- |
Read later
My Reading List
Microsoft Research has released a summary of the results of empirical studies examining software engineering myths. The work, conducted by Nachi Nagappan, measures the impact on quality that common software engineering practices actually have. The analysis reveals:
- More code coverage in testing doesn't necessarily correlate with a decrease in the number of post-release fixes required, citing many other factors that come into play.
TDD improves quality but takes longer (pdf): "What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer." - The use of assertions and code verification decreases bugs. Further, "Software engineers who were able to make productive use of assertions in their code base tended to be well-trained and experienced, a factor that contributed to the end results."
- Organizational structure has a profound impact on quality: "Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent."
- The impact of a team being distributed has a negligible impact on quality.
These research findings are now being used by Microsoft development groups, including helping with risk-analysis and bug-triaging for major projects such as Windows Vista SP2.
Related Sponsor
Agile Project Management That Adapts With You. Provide your global development team a shared space that adapts to the way they work.
No citation about the code coverage section
by
Simon Kirk
Re: No citation about the code coverage section
by
Gavin Terrill
Re: No citation about the code coverage section
by
Jim Leonardo
research.microsoft.com/en-us/projects/esm/nagap...
15 - 35% longer for TDD is misleading
by
Curt Hibbs
So I followed the links to the original paper and found my answer. The 15 - 35% is only the original development time, and does not include the increased maintenance cost of the non-TDD approach. To quote the paper:
Another interesting observation from the outcome measures in Table 3 is the increase in time to develop the features attributed to the usage of the TDD practice, as subjectively estimated by management. The increase in development time ranges from 15% to 35%. From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM.
Re: 15 - 35% longer for TDD is misleading
by
Curt Hibbs
It is incorrect to assert that TDD took 15 to 35% longer. You cannot ignore the rework caused by defects. Its too bad they were not able to quantify this because I'd be willing to bet that this was more than just "offset by reduced maintenance costs" -- that, in fact, there would have been a net increase in productivity for TDD.
Re: 15 - 35% longer for TDD is misleading
by
Sebastian Kübeck
Re: 15 - 35% longer for TDD is misleading
by
Jordi Bieger
What would have happened to the quality if the non-TDD people would have taken that extra 15-35% of time to increase the quality of their code? This would be really interesting to see, because the way the study was done now, the conclusion that spending relatively more time in (initial) development leads to better quality is just as valid as the conclusion that TDD does. It seems intuitive to think that taking that much extra time for refactoring (or simply not hurrying) would (also) improve quality.
Re: 15 - 35% longer for TDD is misleading
by
R Smith
Developer testing and peer review are two of the most effective practices to increase quality. Both of these can provide significant quality gains with some increase in initial development time.
Re: 15 - 35% longer for TDD is misleading
by
Kevin E. Schlabach
Thanks for calling this out because I was definitely thinking it!
Re: 15 - 35% longer for TDD is misleading
by
William Martinez
1. That increase is "subjectively estimated by management", meaning it is a guess.
2. It is an increase in creating the feature.
3. Does this mean TDD coding is added up in the KLOC?
4. If 3, then the density should of course be less!
5. Does this mean non TDD means no unit testing code at all? Because no TDD may mean write the unit test later, not upfront.
6. If 5, then density is also affected. Meaning TDD is either slowing down the production but not increasing the KLOC, or just increasing the KLOC and slowing production too.
I guess the values are too up in the air to mean something I can take a decision with.
William Martinez Pomares
SOme facts about and the research
by
Luca Minudel
Functional tests, pair programming and code reviews are better to decrease bugs number. This is a well known fact and TDD enthusiasts know this. So this research, if I get it right, confirm this.
Also the research state that with TDD it take a longer time (15 to 35 percent longer) to get first release and also that with TDD you reduce post-release maintenance costs significantly. When you consider the first release together with the post-release maintenance, TDD take less time and save costs significantly. This is also a well known fact too and the research confirm this.
The research surface that these are decisions that managers have to make — where should they take the hit.
Imho for a company that use his own software the decision in clear: use TDD, while a sw house can put the maintenance costs on customers (e.g. with maintenance contracts) and decide to have a cheaper lower quality first release: can skip TDD.
Read the linked "Exploding Software-Engineering Myths" post and the linked pdf docs if you want to double-check this.
Impact of org. structure
by
Baljeet Sandhu
Baljeet
Educational Content
Writing Usable APIs in Practice
Giovanni Asproni May 19, 2013
Hello stranger!
You need to Register an InfoQ account or Login to post comments. But there's so much more behind being registered.Get the most out of the InfoQ experience.
Tell us what you think