Opinion: Code Coverage Stats Misleading

Posted by Deborah Hartmann on Aug 30, 2006 03:00 PM

Community

Agile

Topics

Artifacts & Tools,

Unit Testing,

Stories & Case Studies

Tags

Refactoring,

Antipatterns,

Complementary Practices

John Casey,a key contributer to the Apache Software Foundation's Maven Project, recently spent some time refactoring Maven's assembly plugin.  He thought he'd use coverage reporting to mark his testing progress, and to make sure he didn't break anything as he went.  Well, at very least it was a learning experience.

He constructed a completely new suite of tests to focus on small, specific units of the new plugin's implementation.  His refactoring went well, as did the testing.  He had nearly perfect coverage numbers on the parts he felt were critical and felt fairly confident that this plugin would almost drop in as a replacement of the old incarnation.  That's when things started to fall apart.

You can read the details on Casey's blog, along with his reasons for saying: when you're seeking confidence through testing, perhaps the worst thing you can do is to look at a test coverage report.

His conclusion: coverage reporting is dangerous. It has a tendency to distract from the use cases that should drive the software development process. It also tends to give the impression that when the code is covered, it is tested and ready for real life. However, it won't tell you that you've forgotten to test the contract on null handling. It won't tell you that you were supposed to check that String for a leading slash, and trim it off if necessary. To achieve real confidence in your tests, you would have to achieve multiple coverage for most of your code, in order to test each line under various conditions... And, since there is no hard-and-fast rule about how many test passes it will take to test any given line of code adequately, the concept of a coverage report is itself fatally flawed.

1. Back to top

   Look at what isn't covered, not what is

   Aug 30, 2006 5:28 PM by Luke Redpath

   I agree that using code coverage as some kind of measure of how well tested a piece of software is, is flawed. However, whilst there isn't as much value in seeing what *is* covered, I still think it's useful for highlighting what *isnt* covered. Something with 97% code coverage or more isn't necessarily complete, or correct, but its likely to be more reliable than code with low coverage.

   Reply

2. Back to top

   Re: Look at what isn't covered, not what is

   Aug 30, 2006 8:06 PM by Deborah Hartmann

   Umm... isn't the author's point that so-called "97%" code coverage may in fact indicate 10% coverage, if each method has one test but in fact needs 10 to cover basic alternate cases? I'm thinking: wouldn't it be great if we could indicate risk and complexity, and weight those coverage stats?

   Reply

3. Back to top

   Re: Look at what isn't covered, not what is

   Aug 30, 2006 8:52 PM by Scott Battaglia

   While code coverage can't tell you you've covered every possible scenario in terms of input, I've found it to be an indispensible tool in making sure I've taken every *path* in my code.

   Reply

4. Back to top

   Re: Look at what isn't covered, not what is

   Aug 31, 2006 12:53 AM by Steve Bate

   I haven't been able to read the original blog because the site is down, but from this article it appears that some of the confusion is related to the different potential meanings of "coverage". Do you mean 97% code coverage might mean 10% requirements coverage (as in not testing a null handling contract)? If so, that's very true. However, the conclusion that coverage reporting is dangerous is not true. If interpreted properly, it can be a useful tool for identifying holes in testing. I agree that high coverage alone would not give me confidence in the quality of my tests.

   Reply

5. Back to top

   Re: Look at what isn't covered, not what is

   Aug 31, 2006 8:50 AM by Deborah Hartmann

   Do you mean 97% code coverage might mean 10% requirements coverage (as in not testing a null handling contract)?

   Yes. I'm working with acceptance testing right now, so that would be my concern. When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation. If developers have norms about applying good practices for test coverage, they will test well method by method, and then 97% has more meaning for them. If testing is spotty (no agreement among developers on what constitutes adequate coverage of a method) then this is called into question. If there are no norms... it's a crapshoot. Does this sound about right?

   Reply

6. Back to top

   Re: Look at what isn't covered, not what is

   Aug 31, 2006 12:23 PM by Scott Battaglia

   When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.

   We consider those test results internal to our development team and actually never show them to management or clients. We may show them to some of our more "technical" managers, but that's about it.

   Reply

7. Back to top

   Re: Look at what isn't covered, not what is

   Aug 31, 2006 1:36 PM by Deborah Hartmann

   A great idea:

   We consider those test results internal ... never show them to management or clients.

   In the case I experienced, an external QA group was monitoring "coverage" using such figures. Definitely a bad idea. The context of a metric is very important - in the local context it carries with it implicit information that is lost when it's communicated outside.

   Reply

8. Back to top

   Very strange article

   Aug 31, 2006 1:52 PM by Kyrill Alyoshin

   This is getting a bit silly. The only reason I use code coverage is to look what is not covered. And the value of code coverage tools there is tremendous. I'd say unit testing and code coverage go hand-in-hand.

   Reply

9. Back to top

   Re: Very strange article

   Sep 1, 2006 2:54 AM by Paul Oldfield

   Agreed. Code not covered is the interesting information. The question arises, Why isn't it covered? Either there's a missing test, or there's extra code.

   Reply

10. Back to top

    Re: Look at what isn't covered, not what is

    Sep 1, 2006 3:16 AM by Paul Oldfield

    Yes. I'm working with acceptance testing right now, so that would be my concern. When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.

    Test coverage will mean different things depending on what set of tests are run. For Acceptance tests, any uncovered code means that there is a potential that the code may be superfluous. That used to be important when paying by LoC or some derived measure. Of course we wouldn't do that now, would we? Or it might mean we're missing an acceptance test.

    If developers have norms about applying good practices for test coverage, they will test well method by method, and then 97% has more meaning for them. If testing is spotty (no agreement among developers on what constitutes adequate coverage of a method) then this is called into question. If there are no norms... it's a crapshoot.

    For unit testing, coverage can't tell us much useful though it's likely to finger the guy who doesn't write unit tests, or who needs help with them. It might also help to identify 'cruft' left unconnected by poor refactorings. In either case, it's the code not covered that is the interesting information. Looking at what code is covered really doesn't tell us much about how good our testing is, though this is a fairly common beginner's misconception.

    Reply

11. Back to top

    So: poorly named artifact?

    Sep 1, 2006 7:12 AM by Deborah Hartmann

    Paul, it sounds like the moniker "Code Coverage Report" can be one source of confusion: these stats are really intended to indicate code "uncoverage" :-D I'm a stickler for well-named metrics for EXACTLY that reason. Once the thing is out there, people take it at exactly face value - better make sure things are well named!

    Reply

12. Back to top

    Re: So: poorly named artifact?

    Sep 2, 2006 9:21 AM by Paul Oldfield

    Well, it's already named to show code 'covered' not code 'tested'. Off the top of my head, how about "Unexercised Code Report"?

    Reply

13. Back to top

    Limitations of branch and statement coverage

    Sep 4, 2006 8:49 AM by Les Walker

    The limitations of the branch and statement coverage metrics provided by tools like Cobertura and Clover are pretty well known. My -- highly unmathematical -- rule of thumb is that after you get 60% branch or statement coverage, you really need to stop using those metrics and switch to path based metrics. There's good information in Casey's last comment (commenting on your own blog entry?) about testing at different levels. Another rule of thumb that I use is that developers should write white-box tests motivated towards code and domain coverage. QA engineers should write black-box tests motivated towards use-case and feature coverage. Using code coverage metrics as a way of guageing the thoroughness of black-box testing and vice-versa are highly-dubious practices IMO.

    Reply

• Older >