Tracking change and innovation in the enterprise software development community
Free Book Offer
SOA Demystified
$40 Value
QCon London 2010!
March 10-12, 100
Speakers, 80 sessions,
"THE" team leads and,
architects conference.
Scaling Scrum
in the enterprise?
ScrumWorks Pro
can help.
Start a trial today
Posted by Chris Sims on Mar 02, 2009
A paper first published in the Empirical Software Engineering journal reports: "TDD seems to be applicable in various domains and can significantly reduce the defect density of developed software without significant productivity reduction of the development team." The study compared 4 projects, at Microsoft and IBM that used TDD with similar projects that did not use TDD.
The paper was authored by Nachiappan Nagappan (microsoft), E. Michael Maximilien (IBM), Thirumalesh Bhat (Microsoft), and Laurie Williams (North Carolina State University), and published in Volume 13, Number 3 of the journal Emperical Software Engineering. It is also available from the Empirical Software Engineering Group Microsoft Research.
The paper includes 1 case study at IBM and 3 from Microsoft. Each of the case studies compare two teams working on the same product, using the same development languages and technologies, under the same higher-level manager, only one of which was using test-driven development (TDD). None of the teams knew that they would be part of the study during their development cycles. The IBM case study followed teams doing device driver development. The Microsoft cases followed teams working on Windows, MSN, and Visual Studio.
The paper describes the TDD practices used by the teams as minute-to-minute workflows, as well as task-level workflows.
Minute-to-Minute Workflow
Task-Level Workflow
The pre-release defect density of the four products, measured as defects per thousand lines of code, decreased between 40% and 90% relative to the projects that did not use TDD. The teams' management reported subjectively a 15–35% increase in initial development time for the teams using TDD, though the teams agreed that this was offset by reduced maintenance costs.
These results can be compared to those found in a paper published in 2006 by Maria Siniaalto. That paper attempted to review and summarize the results from 13 other studies on test-driven development, including research conducted in industrial, semi-industrial, and academic contexts. Among the conclusions of the paper, the author wrote:
Based on the findings of the existing studies, it can be concluded that TDD seems to improve software quality, especially when employed in an industrial context. The findings were not so obvious in the semiindustrial or academic context, but none of those studies reported on decreased quality either. The productivity effects of TDD were not very obvious, and the results vary regardless of the context of the study. However, there were indications that TDD does not necessarily decrease the developer productivity or extend the project leadtimes: In some cases, significant productivity improvements were achieved with TDD while only two out of thirteen studies reported on decreased productivity. However, in both of those studies the quality was improved.
What are your experiences with TDD? Have you seen an increase in quality? What effects have you seen on developer productivity, and development time? Leave a comment and share your experiences.
Agile Development: A Manager's Roadmap for Success
Learning to be a Good Product Owner: Foundation Skills
Design in Agile, Software Craftsmanship, Product Development, Testing @ QCon London in March
VersionOne is recognized by Agile practitioners as the leader in Agile project management tools. Companies such as Adobe, BBC, CNN, Dow, HP, IBM, Sony and 3M have turned to VersionOne to help deliver greater value to their customers.
I think its fantastic to have a little bit of hard data to back what we practitioners all know as fact, but are so often asked by non-practitioners (for example, all ye PMPs and MBA's) to prove. "Show me the data".
Of course in reading this paper I do question a tad how much the everyone involved in the study *truly* get TDD. But, truth is, it may even be more useful for this to come from "non-expert", non-perfect (from a TDD-sense) angles, being most organizations searching for pudding-proof data aren't likely to have "experts" or perfect situations anyway (who does?) - and judging from this data then, they can still see great benefit!
As a companion to these "hard numbers", I've put together a quick "elevator speech"-ready bullet list of what I most often cite as the primary reasons you can't survive without TDD:
aydsoftware.blogspot.com/2009/03/nutshells-why-...
Cheers
MB
BTW, to be clear, I'm not referring to "ye PMP's and MBA's" in derogatory way - I happen to be of the MBA ilk myself!
I've experienced a similar thing. Generally, a TDD project takes a little more time to ramp up as developers new to TDD get used it and as experienced TDDers figure out what they need to mock.
However, once the project is running, major defects start to shrink away. This leaves your end-of iteration testing to focus on the tests that simply could not be automated. Consequently, this minimizes Quality Assurance resources required for the project.
After a couple of iterations, no one notices TDD as being 'slow'. Instead, everyone comes to love the confidence it gives them. With TDD, there are fewer opportunities to say, "Ahh. I don't really need to test that. It's so straightforward." This kind of anti-TDD thinking adds up and eventually costs.
With TDD, you can realistically achieve extremely high levels of tested code coverage. 70% or more is expected. 95% or higher is very achievable. This results in high levels of confidence in your code base.
In my experience, besides being less error prone, TDD also results in code that is better factored and more maintaintable. Forgot how some esoteric piece of code is supposed to function? Go to the test; it's almost guaranteed to exist.
With TDD, you know the risk of being blindsided by an unintended consequence of your code change is small. This confidence, in turn, reduces the size of estimate you can give for new features.
It's a big win, all around. Glad there's emperical evidence I can now point to, to prove it.
If not for the Microsoft Research Group that made this paper available on their website, 99% of us "industry" folk would not be able to read the paper because Springer charges $34 for this paper, or an unaffordable US$624 per year to subscribe to the journal. It's sad that research so valuable to the community is behind such a high wall.
We've found that in some cases have a TDD environment can reduce developer time. The modify/compile/run cycle is much faster when running a unit test than having to start up a lumbering application to test a small piece of functionality.
I am not here commenting on TDD, which is a fine technique, but on the study, which looks flawed. Here are two salient problems:
1) The team with the lowest code coverage has the best improvement in defect rates. This is precisely the opposite of what TDD suggests. The study's authors do not discuss this.
2) The study is incomplete. The conclusion is that TDD has lower defect rates, but it takes 15% to 35% longer to write the code. If the non-TDD teams were given 15%-35% more time to test and debug their code, what defect density would they have reached? This question is not asked.
Without either question answered, it's hard to really conclude anything about the data.
Brian Marick takes us through a quick tour of the most important values and challenges to adopting Agile successfully (they aren't the typical challenges and values we hear in the community).
The line between development and architecture is tricky. Does it exist at all? Is an ivory tower actually needed? There's a balance in the middle, but how do you move from developer to architect?
The word 'authority' sometimes produces an allergic response in hard-line agilists. Freedom and authority – both are bad if misused and both are good if used in right spirit for a noble cause.
"Getting Started with Grails" brings you up to speed on this modern web framework. Companies as varied as LinkedIn, Wired, and Taco Bell are all using Grails. Are you ready to get started as well?
Those familiar with only ITIL V2 often scoff at the thought that ITIL could serve as a governance framework for SOA. With ITIL V3, the focus of the framework shifted towards service-orientation.
SpringSource CTO Adrian Colyer discusses AspectJ, SpringSource's dm Server and tc Server products, OSGi and Scrum.
Heroku's Adam Wiggins talks about Rails, Background Jobs, Add-Ons, Ruby, and how Heroku manages to work around Ruby's inefficiencies using Erlang and other languages.
For Grady Booch the foundation of a good architecture is patterns, SOA being just one of many patterns. In this Second Life presentation, Booch attempts to bring more clarity on what architecture is.
6 comments
Watch Thread Reply