TDD/BDD Leading To Incomplete Unit Tests?

Posted by Mike Bria on Feb 14, 2008 12:01 AM

Community

Agile

Topics

Methodologies ,

Agile Techniques ,

Object Oriented Design

Tags

TDD ,

Domain Driven Design ,

Fit / Fitnesse ,

XP ,

Testing ,

BDD

Peter Ritchie recently raised concern about what he considers a tendency for adherence to TDD and BDD to keep practitioners from writing good unit tests. In particular, it is the mantra of "interaction testing" that he suggests has the effect of creating incomplete unit tests; tests that fail to show proof that a unit - an object - works under any conditions it could potentially be used. Most interesting about Peter's thoughts may be the counter view they present to the core intentions of TDD and BDD in the first place.

Peter uses the idea of classes being autonomous abstractions of real-world concepts as a foundation for his thoughts that good unit testing is about verifying these autonomous classes and further that TDD and BDD may lead people against this:

One problem I see with Test-Driven Development (TDD) and Behaviour-Driven Development (BDD) is that practitioners simply just center on interaction of the parts of the system and really don't do any "unit testing". They get caught up in the mantras that are TDD and BDD and fail to see the trees for the forest and fall into testing by rote. Unit testing tests individual units, the smallest testable part of an application.

In reference to an example found on Wikipedia's BDD entry, Peter presented the following in support of his case:

The tests detailed only test 13 of 4,294,967,296 possibilities. These tests may very well test the expected behaviour of one system, but don't really test EratosthenesPrimesCalculator as a unit. If the system only allows that behaviour, then these tests prove that it will work. But, if at some point EratosthenesPrimesCalculator is used outside that behaviour (and that's really the purpose of encapsulating code into classes: reuse) not much about EratosthenesPrimesCalculator has been validated.

In large part, the example relies on the thought that a unit's usefulness/correctness is purely an inherent quality of what it's name may imply in the real world. It is this point that many TDD practitioners may challenge, primarily based on the idea that a unit's usefulness is definable only in the context of the environment (system) that is being used. Further, it is the desire to expose these "uses" that TDD/BDD practitioners propose is the primary benefit of the interaction testing Peter refers to in his post. Steve Freeman, co-creator of JMock:

The idea [of test-first interaction testing] is to drive out the relationships that an object has with its environment. For example, you're mocking a DAO, but a DAO is not part of the domain of the application, it's part of the domain of the implementation.

Said another way, many in the TDD school would counter that the primary utility of writing the unit test in the first place is that it is as the explicit specification of what a unit should or should not do. From a discussion comparing TDD to Design By Conctract Mario Gleichmann:

Unit tests as an instrument in the sense of Test Driven Development (TDD) aren’t that much about verification of a correct implementation rather than about a specification of how a unit should behave. in fact, it’s the specification (not the verification), that will drive development. You can see the comeback of this core idea in the rise of Behaviour Driven Development (BDD) that mainly tries to find an adequate vocabulary to write down specifications (that of course can be verified automatically) in an easy natural way, refocussing on how a component should behave under certain conditions.

An often cited corollary to this "a unit is defined by it's context" view is the reminder that unit testing, by definition, does not provide indication to the quality or utility of the system as a whole, and as such should be accompanied during development with adequate levels of acceptance testing. JS Greenwood:

Poor integration tests, as everything is being tested in isolation - we can end up with a system where the constituent parts are clean, isolated, well tested, and known to be correct. But how they fit together is a greyer (or even blacker) area unless [isolated unit testing] is accompanied with a complement of integration tests.

1. Back to top

   Isn't it some kind of continuum?

   Feb 14, 2008 12:25 AM by Eric Torreborre

   My experience of BDD is that specification/verification/testing is a continuum. You start by defining what you want to achieve, then you implement it: that's the specification phase. Then you specify all the implicit/hidden/forgotten assumptions about how your unit should behave: it's the testing phase. This is why in the specs library there are different techniques for specification and testing: - a BDD-like vocabulary:

   "my system" should {
     "behave like this" in {...}
     "behave like that" in {...}
   }
   

   - a ScalaCheck integration to be able to thoroughly test properties and generate a lot of the missing cases. You can find a blog entry here - a JMock integration to isolate the relevant parts. Eric.

   Reply

2. Back to top

   Unit Testing is not Test Driven Design

   Feb 14, 2008 7:32 AM by Mario Gleichmann

   I think we should distinguish the concepts of pure 'Unit Testing' from 'Test Driven Design'(as they are used interchangeably in the post). While (Unit-) Testing is in fact not about specifying the intented behaviour of a component (but ensuring that the affected unit under test works properly under any conceivable situation / condition), Test Driven Design is more about 'Design' than 'Test' (at least initially intended as such). Under this Point of View, BDD is only a 'trend' trying to refocus TDD from 'Test' to 'Design' (resp. specifying the Design by the intended behaviour) I think a good part of the mentioned confusion / complaints is coming from mixing these two ideas. Greetings Mario

   Reply

3. Back to top

   Re: Unit Testing is not Test Driven Design

   Feb 14, 2008 2:02 PM by Erick Dovale

   I couldn't agree more with you Mario. I also fail to see how not testing for boundary cases is in any way related to TDD or BDD for that matter.

   Reply

4. Back to top

   I wish I had this problem

   Feb 14, 2008 5:27 PM by Bruce Rennie

   Not to be flippant but if I had this problem, boy, would I be in a happy place. I can't wait. :)

   Reply

5. Back to top

   Re: Unit Testing is not Test Driven Design

   Feb 15, 2008 10:04 AM by Al Tenhundfeld

   Automated tests are a component of TDD, but automated tests are not necessarily unit tests. BDD/TDD automated tests should focus on the API (external behavior) of a class. They are fairly dumb and ignorant of the class internals. Unit tests should focus on covering all of the code and can be written to test specific aspects of the internal class implementation. That's my general distinction. However, I can understand the article. It's easy to get into the habit of writing your tests, implementing your class, refactoring, etc. and then moving on to the next use case/user story/behavior/whatever. That's part of the reason why code coverage tools are so useful; they can remind you that you neglected unit tests. -Al

   Reply

6. Back to top

   Re: Unit Testing is not Test Driven Design

   Feb 16, 2008 10:59 AM by Michael James

   I have some concerns that unit testing is seen as the be-all and end-all of automated testing. Certainly it's a good start, but failing systems are often built from perfectly good components. more here: http://danube.com/blog/michaeljames/junit_is_not_just_for_unit_testing_anymore and http://danube.com/blog/michaeljames/mock_objects_considered_insufficiently_harmful --mj

   Reply

7. Back to top

   The Wikipedia page

   Feb 18, 2008 6:29 AM by Elizabeth Keogh

   is now updated. Hopefully the new contents may help shed some light.

   Reply

• Older >