Careful With Your Coverage Metrics

Posted by Mike Bria on Nov 12, 2008

Sections

Process & Practices,

Development

Topics

.NET ,

Java ,

Unit Testing ,

Agile Techniques ,

Delivering Quality ,

Agile

Tags

Antipatterns ,

Code Coverage

Christian Gruber takes some time to clarify the TDD stance on using code coverage metrics. He discusses what code coverage metrics do and don't tell you, how TDD fits into the picture, and how one might be best advised to use their code coverage metrics.

Code coverage for an application developed with good TDD will likely be very high (>80-90%). On the other hand, high code coverage on another app says little to nothing about whether that app was built by good TDD, or even TDD at all. Taken further, how well does high code coverage indicate your application is thoroughly tested?

Christian Gruber discusses this, prompted largely by another recent blog post by Kevin Pang on the subject. Out of the gates, Gruber's primary statement is that TDD proponents do not suggest code coverage as "a one true metric", that it is useful but only to degree and taken in context with other sources of feedback. He denounces Pang's claim that "(Pang) 100% code coverage has long been the ultimate goal of testing fanatics", stating that "(Gruber) high code coverage is a desired attribute of a well tested system, but the goal is to have a fully and sufficiently tested system".

He makes the following 6 assertions about code coverage, TDD, and "sufficient testing":

1. Code coverage is only meaningful in the context of well-written tests. It doesn’t save you from crappy tests.
2. Code coverage should only be measured on a line/branch if the covering tests are passing.
3. Code coverage suggests insufficiency, but doesn’t guarantee sufficiency.
4. Test-driven code will likely have the symptom of nearly perfect coverage.
5. Test-driven code will be sufficiently tested, because the author wrote all the tests that form, in full, the requirements/spec of that code.
6. Perfectly covered code will not necessarily be sufficiently tested.

Gruber then expands briefly on how TDD, being a design technique more than a testing tool, helps to provide thorough testing. He further asserts that "code coverage [in the context of TDD] is a great way to notice that you screwed up and missed something, but nothing else", a point that he and Pang seem to largely agree on.

Warning against the misuse of code coverage metrics is not a new thing, although it is a message that needs repeating as more and more organizations are taking on TDD for the first time (congrats!) and can easily fall into the "coverage as gospel" anti-pattern.

For more on this, also see the "Pragmatic Use of Code Coverage Analysis" section in a recent post by Jason Rudolph, which provides a good list of references to other experts' takes on this subject.

1. Back to top

   What kind of coverage?

   by Bruce Rennie

   It pays to know the tools you're using as well. Most of the coverage measurement tools I'm familiar with provide statement or, at best, branch coverage measurements, but not path coverage.
   
   TDD will give you complete path coverage, if you do it properly.

   Reply

2. Back to top

   Re: What kind of coverage?

   by Greg Matthews

   Agree. You should also experience the following with a TDD approach compared to a non-TDD approach.
   
   - More classes, and 'better' classes with higher coherence/lower coupling.
   - Less branches
   - Less code

   Reply

• Older >