How much testing is enough? The answer varies depending on whom you ask. On one end of the spectrum, some say you should strive to achieve 100% test coverage. Others say it doesn't matter, that you should just rely on the quality of the tests, and that measuring test coverage does not tell you anything about the quality of the tests and the code being tested.
Tim Ottinger from ObjectMentor wrote that if you are practicing true TDD then you should have very high test coverage from the fact that you only write production code to satisfy a broken test. The subtle implication is that TDD rarely affects test coverage of existing code.
I’m not saying that code coverage should be low, only that as we move incrementally, each test we write in isolation should have little effect on our code coverage numbers… a thought that intrigues me.
Andy Glover showed, by example, how test coverage metrics can mislead us into a false sense of security. Tests coverage metrics can tell you what code is not tested, but cannot accurately tell you what code is tested. Similarly, Tobias Schlitt argued that code coverage metrics are important because they tell us what parts are not covered.
Surely, a high code coverage rate of a test suite does never indicate, that code is well tested (if you have not written the code and tests yourself). But the other way around works: A small code coverage rate definitely means, that the test suite is not sufficient. But let me dig a bit deeper into code coverage and what it gives you.Testivus, the great testing master, explained it best telling us that "it depends". For someone who is new to testing:
Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.For the experienced developer:
... the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.Finally, for those who simply want answers:
The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.Testing, as described in these blogs focuses on the quality-validation benefits of tests. From this perspective, we should be aware that test-coverage metrics can tell us what is missing much better than they can tell us what is done well.