Rob Windsor on WCF with REST, JSON and RSS
WCF is not just for SOAP based services and can be used with popular protocols like RSS, REST and JSON. Join Rob Windsor as he introduces WCF 3.5 and its new native support for non-SOAP services.
Tracking change and innovation in the enterprise software development community
Posted by Deborah Hartmann on Aug 30, 2006 03:00 PM
John Casey,a key contributer to the Apache Software Foundation's Maven Project, recently spent some time refactoring Maven's assembly plugin. He thought he'd use coverage reporting to mark his testing progress, and to make sure he didn't break anything as he went. Well, at very least it was a learning experience.The Agile Business Analyst: Skills and Techniques needed for Agile
Scaling Agile on large teams & Being Agile every day Tracks @ QCon SF Nov 19-21
I agree that using code coverage as some kind of measure of how well tested a piece of software is, is flawed. However, whilst there isn't as much value in seeing what *is* covered, I still think it's useful for highlighting what *isnt* covered. Something with 97% code coverage or more isn't necessarily complete, or correct, but its likely to be more reliable than code with low coverage.
Umm... isn't the author's point that so-called "97%" code coverage may in fact indicate 10% coverage, if each method has one test but in fact needs 10 to cover basic alternate cases? I'm thinking: wouldn't it be great if we could indicate risk and complexity, and weight those coverage stats?
While code coverage can't tell you you've covered every possible scenario in terms of input, I've found it to be an indispensible tool in making sure I've taken every *path* in my code.
I haven't been able to read the original blog because the site is down, but from this article it appears that some of the confusion is related to the different potential meanings of "coverage". Do you mean 97% code coverage might mean 10% requirements coverage (as in not testing a null handling contract)? If so, that's very true. However, the conclusion that coverage reporting is dangerous is not true. If interpreted properly, it can be a useful tool for identifying holes in testing. I agree that high coverage alone would not give me confidence in the quality of my tests.
Do you mean 97% code coverage might mean 10% requirements coverage (as in not testing a null handling contract)?
Yes. I'm working with acceptance testing right now, so that would be my concern. When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.
If developers have norms about applying good practices for test coverage, they will test well method by method, and then 97% has more meaning for them. If testing is spotty (no agreement among developers on what constitutes adequate coverage of a method) then this is called into question. If there are no norms... it's a crapshoot.
Does this sound about right?
When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.
We consider those test results internal to our development team and actually never show them to management or clients. We may show them to some of our more "technical" managers, but that's about it.
A great idea:We consider those test results internal ... never show them to management or clients.
In the case I experienced, an external QA group was monitoring "coverage" using such figures. Definitely a bad idea.
The context of a metric is very important - in the local context it carries with it implicit information that is lost when it's communicated outside.
This is getting a bit silly. The only reason I use code coverage is to look what is not covered. And the value of code coverage tools there is tremendous. I'd say unit testing and code coverage go hand-in-hand.
Agreed. Code not covered is the interesting information. The question arises, Why isn't it covered? Either there's a missing test, or there's extra code.
Yes. I'm working with acceptance testing right now, so that would be my concern. When people other than developers look at these reports, they are unfortunately misinterpreted, because the number looks simple and solid but is really in need of interpretation.
Test coverage will mean different things depending on what set of tests are run. For Acceptance tests, any uncovered code means that there is a potential that the code may be superfluous. That used to be important when paying by LoC or some derived measure. Of course we wouldn't do that now, would we? Or it might mean we're missing an acceptance test.
If developers have norms about applying good practices for test coverage, they will test well method by method, and then 97% has more meaning for them. If testing is spotty (no agreement among developers on what constitutes adequate coverage of a method) then this is called into question. If there are no norms... it's a crapshoot.
For unit testing, coverage can't tell us much useful though it's likely to finger the guy who doesn't write unit tests, or who needs help with them. It might also help to identify 'cruft' left unconnected by poor refactorings.
In either case, it's the code not covered that is the interesting information. Looking at what code is covered really doesn't tell us much about how good our testing is, though this is a fairly common beginner's misconception.
Paul, it sounds like the moniker "Code Coverage Report" can be one source of confusion: these stats are really intended to indicate code "uncoverage" :-D I'm a stickler for well-named metrics for EXACTLY that reason. Once the thing is out there, people take it at exactly face value - better make sure things are well named!
Well, it's already named to show code 'covered' not code 'tested'. Off the top of my head, how about "Unexercised Code Report"?
The limitations of the branch and statement coverage metrics provided by tools like Cobertura and Clover are pretty well known. My -- highly unmathematical -- rule of thumb is that after you get 60% branch or statement coverage, you really need to stop using those metrics and switch to path based metrics. There's good information in Casey's last comment (commenting on your own blog entry?) about testing at different levels. Another rule of thumb that I use is that developers should write white-box tests motivated towards code and domain coverage. QA engineers should write black-box tests motivated towards use-case and feature coverage. Using code coverage metrics as a way of guageing the thoroughness of black-box testing and vice-versa are highly-dubious practices IMO.
WCF is not just for SOAP based services and can be used with popular protocols like RSS, REST and JSON. Join Rob Windsor as he introduces WCF 3.5 and its new native support for non-SOAP services.
Christophe Coenraets discusses Flex 3, Flex Builder, AIR, BlazeDS, Adobe and open source, integrating Flex with existing applications, and integrating RIAs with search engines and browsers.
Danijel Arsenovski attempts to dispel some of the myths around refactoring and how it applies to .NET developers.
In this presentation, recorded at QCon San Francisco, CORBA guru Steve Vinoski explains REST from the view of someone who comes to SOA from a traditional, RPC-oriented background.
Feature teams are key to scaling agility for large teams. In an excerpt from "Scaling Lean and Agile Development," Larman & Vodde show how feature teams resolve traditional problems & raise new issues
Billy Newport talks about virtualization, eXtreme Transaction Processing (XTP) and WebSphere Virtual Enterprise. He discusses hardware, hypervisor, JVM, application and data virtualization.
While virtualization provides many benefits, security can not be a forgotten concept in its application.
This session is specifically aimed at traditionally trained project managers who are new to Agile, and who would like to be able to relate the PMI's best practices to their Agile equivalents.
13 comments
Reply