100% Test Coverage?
How much testing is enough? The answer varies depending on whom you ask. On one end of the spectrum, some say you should strive to achieve 100% test coverage. Others say it doesn't matter, that you should just rely on the quality of the tests, and that measuring test coverage does not tell you anything about the quality of the tests and the code being tested.
Tim Ottinger from ObjectMentor wrote that if you are practicing true TDD then you should have very high test coverage from the fact that you only write production code to satisfy a broken test. The subtle implication is that TDD rarely affects test coverage of existing code.
I’m not saying that code coverage should be low, only that as we move incrementally, each test we write in isolation should have little effect on our code coverage numbers… a thought that intrigues me.
Andy Glover showed, by example, how test coverage metrics can mislead us into a false sense of security. Tests coverage metrics can tell you what code is not tested, but cannot accurately tell you what code is tested. Similarly, Tobias Schlitt argued that code coverage metrics are important because they tell us what parts are not covered.
Surely, a high code coverage rate of a test suite does never indicate, that code is well tested (if you have not written the code and tests yourself). But the other way around works: A small code coverage rate definitely means, that the test suite is not sufficient. But let me dig a bit deeper into code coverage and what it gives you.Testivus, the great testing master, explained it best telling us that "it depends". For someone who is new to testing:
Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.For the experienced developer:
... the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.Finally, for those who simply want answers:
The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.Testing, as described in these blogs focuses on the quality-validation benefits of tests. From this perspective, we should be aware that test-coverage metrics can tell us what is missing much better than they can tell us what is done well.
All it calls for is some common sense, use your atomic/unit tests to provide rapid feedback and design, and higher level (integration/functional/acceptance) tests to verify that your application is behaving as a whole. How much effort you put in to each of these levels depends on the application you're developing and the risk involved.
In my opinion, although 100% code coverage does not mean well tested code, it does tell us an important metric that there is no redundant code. I agree that we still have to make sure all the branching coverage is very high as well and we cover all the corner cases and the functional as well as integration tests using something like Cactus. Typically the person who wrote the class is the best to write the test cases as well as they know all the corner cases the best.
Thanks for your link. Nice post there.
Then, of course, there is the fact that good tests change the way a developer approaches and solves a problem. It encourages very loosely coupled code and cohesive classes. This is probably more important than the validation features of tests and shows up with test-first development.
I'd prefer we never mentioned coverage at all.
1. If your coverage is low but there are few reported bugs, your customers are happy, releases are solid, money is flowing, etc. Do you start writing new tests?
2. If your coverage is high, but there are many reported defects, your customers are unhappy, the team is demoralized, etc. Do you point to the coverage numbers and say "but it's all tested"?
Coverage can't actually help me make those macro level decisions. All coverage can do is tell me where I should invest my testing dollar IF (and only if) I've already decided I have a problem.
The only reason we talk about coverage as much as we do is that it's become relatively easy to measure.
Brandon Holt, Preston Briggs, Luis Ceze, Mark Oskin May 21, 2015
Kai Kreuzer, Olaf Weinmann May 21, 2015