Empirical Studies Show Test Driven Development Improves Quality

Posted by Chris Sims on Mar 02, 2009

Community

Agile

Topics

Agile Techniques

Tags

TDD

A paper first published in the Empirical Software Engineering journal reports: "TDD seems to be applicable in various domains and can significantly reduce the defect density of developed software without significant productivity reduction of the development team." The study compared 4 projects, at Microsoft and IBM that used TDD with similar projects that did not use TDD.

The paper was authored by Nachiappan Nagappan (microsoft), E. Michael Maximilien (IBM), Thirumalesh Bhat (Microsoft), and Laurie Williams (North Carolina State University), and published in Volume 13, Number 3 of the journal Emperical Software Engineering. It is also available from the Empirical Software Engineering Group Microsoft Research.

The paper includes 1 case study at IBM and 3 from Microsoft. Each of the case studies compare two teams working on the same product, using the same development languages and technologies, under the same higher-level manager, only one of which was using test-driven development (TDD). None of the teams knew that they would be part of the study during their development cycles. The IBM case study followed teams doing device driver development. The Microsoft cases followed teams working on Windows, MSN, and Visual Studio.

The paper describes the TDD practices used by the teams as minute-to-minute workflows, as well as task-level workflows.

Minute-to-Minute Workflow

• Write a small number of new tests
• Run the tests and see that they fail
• Implement code to satisfy the tests
• Re-run the new unit test cases to ensure they now pass

Task-Level Workflow

• Integrate new code and tests into the existing code base
• Re-run all the test cases to ensure the new code does not break anything
• Refactor the implementation and/or test code
• Re-run all tests to ensure that the refactored code does not break anything

The pre-release defect density of the four products, measured as defects per thousand lines of code, decreased between 40% and 90% relative to the projects that did not use TDD. The teams' management reported subjectively a 15–35% increase in initial development time for the teams using TDD, though the teams agreed that this was offset by reduced maintenance costs.

These results can be compared to those found in a paper published in 2006 by Maria Siniaalto. That paper attempted to review and summarize the results from 13 other studies on test-driven development, including research conducted in industrial, semi-industrial, and academic contexts. Among the conclusions of the paper, the author wrote:

Based on the findings of the existing studies, it can be concluded that TDD seems to improve software quality, especially when employed in an industrial context. The findings were not so obvious in the semiindustrial or academic context, but none of those studies reported on decreased quality either. The productivity effects of TDD were not very obvious, and the results vary regardless of the context of the study. However, there were indications that TDD does not necessarily decrease the developer productivity or extend the project leadtimes: In some cases, significant productivity improvements were achieved with TDD while only two out of thirteen studies reported on decreased productivity. However, in both of those studies the quality was improved.

What are your experiences with TDD? Have you seen an increase in quality? What effects have you seen on developer productivity, and development time? Leave a comment and share your experiences.

1. Back to top

   Fantastic

   Mar 2, 2009 10:48 PM by Mike Bria

   I think its fantastic to have a little bit of hard data to back what we practitioners all know as fact, but are so often asked by non-practitioners (for example, all ye PMPs and MBA's) to prove. "Show me the data".
   
   Of course in reading this paper I do question a tad how much the everyone involved in the study *truly* get TDD. But, truth is, it may even be more useful for this to come from "non-expert", non-perfect (from a TDD-sense) angles, being most organizations searching for pudding-proof data aren't likely to have "experts" or perfect situations anyway (who does?) - and judging from this data then, they can still see great benefit!
   
   As a companion to these "hard numbers", I've put together a quick "elevator speech"-ready bullet list of what I most often cite as the primary reasons you can't survive without TDD:
   aydsoftware.blogspot.com/2009/03/nutshells-why-...
   
   Cheers
   MB

   Reply

2. Back to top

   Re: Fantastic

   Mar 3, 2009 7:19 AM by Mike Bria

   BTW, to be clear, I'm not referring to "ye PMP's and MBA's" in derogatory way - I happen to be of the MBA ilk myself!

   Reply

3. Back to top

   Reflects My Experience

   Mar 3, 2009 11:02 AM by Benjamin Booth

   I've experienced a similar thing. Generally, a TDD project takes a little more time to ramp up as developers new to TDD get used it and as experienced TDDers figure out what they need to mock.
   
   However, once the project is running, major defects start to shrink away. This leaves your end-of iteration testing to focus on the tests that simply could not be automated. Consequently, this minimizes Quality Assurance resources required for the project.
   
   After a couple of iterations, no one notices TDD as being 'slow'. Instead, everyone comes to love the confidence it gives them. With TDD, there are fewer opportunities to say, "Ahh. I don't really need to test that. It's so straightforward." This kind of anti-TDD thinking adds up and eventually costs.
   
   With TDD, you can realistically achieve extremely high levels of tested code coverage. 70% or more is expected. 95% or higher is very achievable. This results in high levels of confidence in your code base.
   
   In my experience, besides being less error prone, TDD also results in code that is better factored and more maintaintable. Forgot how some esoteric piece of code is supposed to function? Go to the test; it's almost guaranteed to exist.
   
   With TDD, you know the risk of being blindsided by an unintended consequence of your code change is small. This confidence, in turn, reduces the size of estimate you can give for new features.
   
   It's a big win, all around. Glad there's emperical evidence I can now point to, to prove it.

   Reply

4. Back to top

   Research is too expensive to access

   Mar 3, 2009 1:29 PM by Ted Young

   If not for the Microsoft Research Group that made this paper available on their website, 99% of us "industry" folk would not be able to read the paper because Springer charges $34 for this paper, or an unaffordable US$624 per year to subscribe to the journal. It's sad that research so valuable to the community is behind such a high wall.

   Reply

5. Back to top

   Reduced developer time.

   Mar 3, 2009 2:07 PM by Felix Collins

   We've found that in some cases have a TDD environment can reduce developer time. The modify/compile/run cycle is much faster when running a unit test than having to start up a lumbering application to test a small piece of functionality.

   Reply

6. Back to top

   Deeply flawed

   Nov 6, 2009 3:53 PM by Mister Bean

   I am not here commenting on TDD, which is a fine technique, but on the study, which looks flawed. Here are two salient problems:
   
   1) The team with the lowest code coverage has the best improvement in defect rates. This is precisely the opposite of what TDD suggests. The study's authors do not discuss this.
   
   2) The study is incomplete. The conclusion is that TDD has lower defect rates, but it takes 15% to 35% longer to write the code. If the non-TDD teams were given 15%-35% more time to test and debug their code, what defect density would they have reached? This question is not asked.
   
   Without either question answered, it's hard to really conclude anything about the data.

   Reply

• Older >