Tracking change and innovation in the enterprise software development community
QCon London 2010!
March 10-12, 100
Speakers, 80 sessions,
"THE" team leads and,
architects conference.
Free Book Offer
SOA Demystified
$40 Value
Scaling Scrum
in the enterprise?
ScrumWorks Pro
can help.
Start a trial today
Posted by Gavin Terrill on Oct 08, 2009
Microsoft Research has released a summary of the results of empirical studies examining software engineering myths. The work, conducted by Nachi Nagappan, measures the impact on quality that common software engineering practices actually have. The analysis reveals:
These research findings are now being used by Microsoft development groups, including helping with risk-analysis and bug-triaging for major projects such as Windows Vista SP2.
Agile Development: A Manager's Roadmap for Success
Right Collaboration Architecture Drives Business Transformation
VersionOne is recognized by Agile practitioners as the leader in Agile project management tools. Companies such as Adobe, BBC, CNN, Dow, HP, IBM, Sony and 3M have turned to VersionOne to help deliver greater value to their customers.
I'm wondering why there's no citation on the assertion that "More code coverage in testing doesn't necessarily correlate with a decrease in the number of post-release fixes required, citing many other factors that come into play"?
Yes, not sure about that. I did find this link on the microsoft research site - it looks like something might be available through IEEE.
I believe you'll find that in the links on the right of the MS article. This was an older paper they had put out (about 1 1/2 yrs ago).
research.microsoft.com/en-us/projects/esm/nagap...
I wondered how TDD could report 60 to 90% fewer defects but still take 15 to 35% longer, since fewer defects means less rework. It didn't add up, logically.
So I followed the links to the original paper and found my answer. The 15 - 35% is only the original development time, and does not include the increased maintenance cost of the non-TDD approach. To quote the paper:Another interesting observation from the outcome measures in Table 3 is the increase in time to develop the features attributed to the usage of the TDD practice, as subjectively estimated by management. The increase in development time ranges from 15% to 35%. From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM.
I guess I'd like to state this explicitly:
It is incorrect to assert that TDD took 15 to 35% longer. You cannot ignore the rework caused by defects. Its too bad they were not able to quantify this because I'd be willing to bet that this was more than just "offset by reduced maintenance costs" -- that, in fact, there would have been a net increase in productivity for TDD.
Quoting Kent Beck: “If I don’t need to make it work, I can go a lot faster.”
What is the difference between "more reworking needs to be done" and "increased maintenance costs"? Anyway, I was wondering the exact opposite from you:
What would have happened to the quality if the non-TDD people would have taken that extra 15-35% of time to increase the quality of their code? This would be really interesting to see, because the way the study was done now, the conclusion that spending relatively more time in (initial) development leads to better quality is just as valid as the conclusion that TDD does. It seems intuitive to think that taking that much extra time for refactoring (or simply not hurrying) would (also) improve quality.
The most likely outcome of simply allocating 15-35% more time to development will not be an increase in quality, but an increase in features. To increase quality, it is the process that needs to be changed, not just the time.
Developer testing and peer review are two of the most effective practices to increase quality. Both of these can provide significant quality gains with some increase in initial development time.
This makes sense coming from Microsoft... their products are clearly delivered before the bugs are found and removed. They never account for that in their "total lifecycle" and it appears their whole project is just the development time.
Thanks for calling this out because I was definitely thinking it!
Wait, not that simple.
1. That increase is "subjectively estimated by management", meaning it is a guess.
2. It is an increase in creating the feature.
3. Does this mean TDD coding is added up in the KLOC?
4. If 3, then the density should of course be less!
5. Does this mean non TDD means no unit testing code at all? Because no TDD may mean write the unit test later, not upfront.
6. If 5, then density is also affected. Meaning TDD is either slowing down the production but not increasing the KLOC, or just increasing the KLOC and slowing production too.
I guess the values are too up in the air to mean something I can take a decision with.
William Martinez Pomares
TDD is supposed to improve code quality (code easy to understand, change, evolve; higher team velocity; effort estimations easier).
Functional tests, pair programming and code reviews are better to decrease bugs number. This is a well known fact and TDD enthusiasts know this. So this research, if I get it right, confirm this.
Also the research state that with TDD it take a longer time (15 to 35 percent longer) to get first release and also that with TDD you reduce post-release maintenance costs significantly. When you consider the first release together with the post-release maintenance, TDD take less time and save costs significantly. This is also a well known fact too and the research confirm this.
The research surface that these are decisions that managers have to make — where should they take the hit.
Imho for a company that use his own software the decision in clear: use TDD, while a sw house can put the maintenance costs on customers (e.g. with maintenance contracts) and decide to have a cheaper lower quality first release: can skip TDD.
Read the linked "Exploding Software-Engineering Myths" post and the linked pdf docs if you want to double-check this.
I've always loved this quote!
From the paper on the the its not very clear (to me at least) what the recommendations are. I am still grappling with the formula they provide in section 5.3. I agree that org. structure impacts software quality but the question then is, what do the authors recommend as a good org. structure conductive to software dev. ?
Baljeet
Brian Marick takes us through a quick tour of the most important values and challenges to adopting Agile successfully (they aren't the typical challenges and values we hear in the community).
The line between development and architecture is tricky. Does it exist at all? Is an ivory tower actually needed? There's a balance in the middle, but how do you move from developer to architect?
The word 'authority' sometimes produces an allergic response in hard-line agilists. Freedom and authority – both are bad if misused and both are good if used in right spirit for a noble cause.
"Getting Started with Grails" brings you up to speed on this modern web framework. Companies as varied as LinkedIn, Wired, and Taco Bell are all using Grails. Are you ready to get started as well?
Those familiar with only ITIL V2 often scoff at the thought that ITIL could serve as a governance framework for SOA. With ITIL V3, the focus of the framework shifted towards service-orientation.
SpringSource CTO Adrian Colyer discusses AspectJ, SpringSource's dm Server and tc Server products, OSGi and Scrum.
Heroku's Adam Wiggins talks about Rails, Background Jobs, Add-Ons, Ruby, and how Heroku manages to work around Ruby's inefficiencies using Erlang and other languages.
For Grady Booch the foundation of a good architecture is patterns, SOA being just one of many patterns. In this Second Life presentation, Booch attempts to bring more clarity on what architecture is.
13 comments
Watch Thread Reply