Careful With Your Coverage Metrics
Christian Gruber takes some time to clarify the TDD stance on using code coverage metrics. He discusses what code coverage metrics do and don't tell you, how TDD fits into the picture, and how one might be best advised to use their code coverage metrics.
Code coverage for an application developed with good TDD will likely be very high (>80-90%). On the other hand, high code coverage on another app says little to nothing about whether that app was built by good TDD, or even TDD at all. Taken further, how well does high code coverage indicate your application is thoroughly tested?
Christian Gruber discusses this, prompted largely by another recent blog post by Kevin Pang on the subject. Out of the gates, Gruber's primary statement is that TDD proponents do
He makes the following 6 assertions about code coverage, TDD, and "sufficient testing":
- Code coverage is only meaningful in the context of well-written tests. It doesn’t save you from crappy tests.
- Code coverage should only be measured on a line/branch if the covering tests are passing.
- Code coverage suggests insufficiency, but doesn’t guarantee sufficiency.
- Test-driven code will likely have the symptom of nearly perfect coverage.
- Test-driven code will be sufficiently tested, because the author wrote all the tests that form, in full, the requirements/spec of that code.
- Perfectly covered code will not necessarily be sufficiently tested.
Gruber then expands briefly on how TDD, being a design technique more than a testing tool, helps to provide thorough testing. He further asserts that "code coverage [in the context of TDD] is a great way to notice that you screwed up and missed something, but nothing else", a point that he and Pang seem to largely agree on.
Warning against the misuse of code coverage metrics is not a new thing, although it is a message that needs repeating as more and more organizations are taking on TDD for the first time (congrats!) and can easily fall into the "coverage as gospel" anti-pattern.
For more on this, also see the "Pragmatic Use of Code Coverage Analysis" section in a recent post by Jason Rudolph, which provides a good list of references to other experts' takes on this subject.
What kind of coverage?
TDD will give you complete path coverage, if you do it properly.
Roy Rapoport Aug 28, 2014