BT

Careful With Your Coverage Metrics

by Mike Bria on Nov 12, 2008 |

Christian Gruber takes some time to clarify the TDD stance on using code coverage metrics. He discusses what code coverage metrics do and don't tell you, how TDD fits into the picture, and how one might be best advised to use their code coverage metrics.

Code coverage for an application developed with good TDD will likely be very high (>80-90%). On the other hand, high code coverage on another app says little to nothing about whether that app was built by good TDD, or even TDD at all. Taken further, how well does high code coverage indicate your application is thoroughly tested?

Christian Gruber discusses this, prompted largely by another recent blog post by Kevin Pang on the subject. Out of the gates, Gruber's primary statement is that TDD proponents do not suggest code coverage as "a one true metric", that it is useful but only to degree and taken in context with other sources of feedback. He denounces Pang's claim that "(Pang) 100% code coverage has long been the ultimate goal of testing fanatics", stating that "(Gruber) high code coverage is a desired attribute of a well tested system, but the goal is to have a fully and sufficiently tested system".

He makes the following 6 assertions about code coverage, TDD, and "sufficient testing":

  1. Code coverage is only meaningful in the context of well-written tests. It doesn’t save you from crappy tests.
  2. Code coverage should only be measured on a line/branch if the covering tests are passing.
  3. Code coverage suggests insufficiency, but doesn’t guarantee sufficiency.
  4. Test-driven code will likely have the symptom of nearly perfect coverage.
  5. Test-driven code will be sufficiently tested, because the author wrote all the tests that form, in full, the requirements/spec of that code.
  6. Perfectly covered code will not necessarily be sufficiently tested.

Gruber then expands briefly on how TDD, being a design technique more than a testing tool, helps to provide thorough testing. He further asserts that "code coverage [in the context of TDD] is a great way to notice that you screwed up and missed something, but nothing else", a point that he and Pang seem to largely agree on.

Warning against the misuse of code coverage metrics is not a new thing, although it is a message that needs repeating as more and more organizations are taking on TDD for the first time (congrats!) and can easily fall into the "coverage as gospel" anti-pattern.

For more on this, also see the "Pragmatic Use of Code Coverage Analysis" section in a recent post by Jason Rudolph, which provides a good list of references to other experts' takes on this subject.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

What kind of coverage? by Bruce Rennie

It pays to know the tools you're using as well. Most of the coverage measurement tools I'm familiar with provide statement or, at best, branch coverage measurements, but not path coverage.

TDD will give you complete path coverage, if you do it properly.

Re: What kind of coverage? by Greg Matthews

Agree. You should also experience the following with a TDD approach compared to a non-TDD approach.

- More classes, and 'better' classes with higher coherence/lower coupling.
- Less branches
- Less code

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

2 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT