Tracking change and innovation in the enterprise software development community
Free Book Offer
SOA Demystified
$40 Value
Scaling Scrum
in the enterprise?
ScrumWorks Pro
can help.
Start a trial today
QCon London 2010!
March 10-12, 100
Speakers, 80 sessions,
"THE" team leads and,
architects conference.
Posted by Deborah Hartmann Preuss on Aug 30, 2006
John Casey,a key contributer to the Apache Software Foundation's Maven Project, recently spent some time refactoring Maven's assembly plugin. He thought he'd use coverage reporting to mark his testing progress, and to make sure he didn't break anything as he went. Well, at very least it was a learning experience.Right Collaboration Architecture Drives Business Transformation
Learning to be a Good Product Owner: Foundation Skills
Technical Debt and Design Death
VersionOne is recognized by Agile practitioners as the leader in Agile project management tools. Companies such as Adobe, BBC, CNN, Dow, HP, IBM, Sony and 3M have turned to VersionOne to help deliver greater value to their customers.
I agree that using code coverage as some kind of measure of how well tested a piece of software is, is flawed.
However, whilst there isn't as much value in seeing what *is* covered, I still think it's useful for highlighting what *isnt* covered. Something with 97% code coverage or more isn't necessarily complete, or correct, but its likely to be more reliable than code with low coverage.
Umm... isn't the author's point that so-called "97%" code coverage may in fact indicate 10% coverage, if each method has one test but in fact needs 10 to cover basic alternate cases?
I'm thinking: wouldn't it be great if we could indicate risk and complexity, and weight those coverage stats?
While code coverage can't tell you you've covered every possible scenario in terms of input, I've found it to be an indispensible tool in making sure I've taken every *path* in my code.
I haven't been able to read the original blog because the site is down, but from this article it appears that some of the confusion is related to the different potential meanings of "coverage". Do you mean 97% code coverage might mean 10% requirements coverage (as in not testing a null handling contract)? If so, that's very true. However, the conclusion that coverage reporting is dangerous is not true. If interpreted properly, it can be a useful tool for identifying holes in testing. I agree that high coverage alone would not give me confidence in the quality of my tests.
Do you mean 97% code coverage might mean 10% requirements coverage (as in not testing a null handling contract)?
Yes. I'm working with acceptance testing right now, so that would be my concern. When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.
If developers have norms about applying good practices for test coverage, they will test well method by method, and then 97% has more meaning for them. If testing is spotty (no agreement among developers on what constitutes adequate coverage of a method) then this is called into question. If there are no norms... it's a crapshoot.
Does this sound about right?
When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.
We consider those test results internal to our development team and actually never show them to management or clients. We may show them to some of our more "technical" managers, but that's about it.
A great idea:We consider those test results internal ... never show them to management or clients.
In the case I experienced, an external QA group was monitoring "coverage" using such figures. Definitely a bad idea.
The context of a metric is very important - in the local context it carries with it implicit information that is lost when it's communicated outside.
This is getting a bit silly. The only reason I use code coverage is to look what is not covered. And the value of code coverage tools there is tremendous. I'd say unit testing and code coverage go hand-in-hand.
Agreed. Code not covered is the interesting information. The question arises, Why isn't it covered? Either there's a missing test, or there's extra code.
Yes. I'm working with acceptance testing right now, so that would be my concern. When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.
Test coverage will mean different things depending on what set of tests are run. For Acceptance tests, any uncovered code means that there is a potential that the code may be superfluous. That used to be important when paying by LoC or some derived measure. Of course we wouldn't do that now, would we? Or it might mean we're missing an acceptance test.If developers have norms about applying good practices for test coverage, they will test well method by method, and then 97% has more meaning for them. If testing is spotty (no agreement among developers on what constitutes adequate coverage of a method) then this is called into question. If there are no norms... it's a crapshoot.
For unit testing, coverage can't tell us much useful though it's likely to finger the guy who doesn't write unit tests, or who needs help with them. It might also help to identify 'cruft' left unconnected by poor refactorings.
In either case, it's the code not covered that is the interesting information. Looking at what code is covered really doesn't tell us much about how good our testing is, though this is a fairly common beginner's misconception.
Paul, it sounds like the moniker "Code Coverage Report" can be one source of confusion: these stats are really intended to indicate code "uncoverage" :-D
I'm a stickler for well-named metrics for EXACTLY that reason. Once the thing is out there, people take it at exactly face value - better make sure things are well named!
Well, it's already named to show code 'covered' not code 'tested'. Off the top of my head, how about "Unexercised Code Report"?
The limitations of the branch and statement coverage metrics provided by tools like Cobertura and Clover are pretty well known. My -- highly unmathematical -- rule of thumb is that after you get 60% branch or statement coverage, you really need to stop using those metrics and switch to path based metrics.
There's good information in Casey's last comment (commenting on your own blog entry?) about testing at different levels. Another rule of thumb that I use is that developers should write white-box tests motivated towards code and domain coverage. QA engineers should write black-box tests motivated towards use-case and feature coverage. Using code coverage metrics as a way of guageing the thoroughness of black-box testing and vice-versa are highly-dubious practices IMO.
Brian Marick takes us through a quick tour of the most important values and challenges to adopting Agile successfully (they aren't the typical challenges and values we hear in the community).
The line between development and architecture is tricky. Does it exist at all? Is an ivory tower actually needed? There's a balance in the middle, but how do you move from developer to architect?
The word 'authority' sometimes produces an allergic response in hard-line agilists. Freedom and authority – both are bad if misused and both are good if used in right spirit for a noble cause.
"Getting Started with Grails" brings you up to speed on this modern web framework. Companies as varied as LinkedIn, Wired, and Taco Bell are all using Grails. Are you ready to get started as well?
Those familiar with only ITIL V2 often scoff at the thought that ITIL could serve as a governance framework for SOA. With ITIL V3, the focus of the framework shifted towards service-orientation.
SpringSource CTO Adrian Colyer discusses AspectJ, SpringSource's dm Server and tc Server products, OSGi and Scrum.
Heroku's Adam Wiggins talks about Rails, Background Jobs, Add-Ons, Ruby, and how Heroku manages to work around Ruby's inefficiencies using Erlang and other languages.
For Grady Booch the foundation of a good architecture is patterns, SOA being just one of many patterns. In this Second Life presentation, Booch attempts to bring more clarity on what architecture is.
13 comments
Watch Thread Reply