BT

Your opinion matters! Please fill in the InfoQ Survey!

Careful With Your Coverage Metrics

| by Mike Bria Follow 0 Followers on Nov 12, 2008. Estimated reading time: 2 minutes |

A note to our readers: As per your request we have developed a set of features that allow you to reduce the noise, while not losing sight of anything that is important. Get email and web notifications by choosing the topics you are interested in.

Christian Gruber takes some time to clarify the TDD stance on using code coverage metrics. He discusses what code coverage metrics do and don't tell you, how TDD fits into the picture, and how one might be best advised to use their code coverage metrics.

Code coverage for an application developed with good TDD will likely be very high (>80-90%). On the other hand, high code coverage on another app says little to nothing about whether that app was built by good TDD, or even TDD at all. Taken further, how well does high code coverage indicate your application is thoroughly tested?

Christian Gruber discusses this, prompted largely by another recent blog post by Kevin Pang on the subject. Out of the gates, Gruber's primary statement is that TDD proponents do not suggest code coverage as "a one true metric", that it is useful but only to degree and taken in context with other sources of feedback. He denounces Pang's claim that "(Pang) 100% code coverage has long been the ultimate goal of testing fanatics", stating that "(Gruber) high code coverage is a desired attribute of a well tested system, but the goal is to have a fully and sufficiently tested system".

He makes the following 6 assertions about code coverage, TDD, and "sufficient testing":

  1. Code coverage is only meaningful in the context of well-written tests. It doesn’t save you from crappy tests.
  2. Code coverage should only be measured on a line/branch if the covering tests are passing.
  3. Code coverage suggests insufficiency, but doesn’t guarantee sufficiency.
  4. Test-driven code will likely have the symptom of nearly perfect coverage.
  5. Test-driven code will be sufficiently tested, because the author wrote all the tests that form, in full, the requirements/spec of that code.
  6. Perfectly covered code will not necessarily be sufficiently tested.

Gruber then expands briefly on how TDD, being a design technique more than a testing tool, helps to provide thorough testing. He further asserts that "code coverage [in the context of TDD] is a great way to notice that you screwed up and missed something, but nothing else", a point that he and Pang seem to largely agree on.

Warning against the misuse of code coverage metrics is not a new thing, although it is a message that needs repeating as more and more organizations are taking on TDD for the first time (congrats!) and can easily fall into the "coverage as gospel" anti-pattern.

For more on this, also see the "Pragmatic Use of Code Coverage Analysis" section in a recent post by Jason Rudolph, which provides a good list of references to other experts' takes on this subject.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

What kind of coverage? by Bruce Rennie

It pays to know the tools you're using as well. Most of the coverage measurement tools I'm familiar with provide statement or, at best, branch coverage measurements, but not path coverage.

TDD will give you complete path coverage, if you do it properly.

Re: What kind of coverage? by Greg Matthews

Agree. You should also experience the following with a TDD approach compared to a non-TDD approach.

- More classes, and 'better' classes with higher coherence/lower coupling.
- Less branches
- Less code

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

2 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT