TDD Opinion: Quality Is a Function of Thought and Reflection, Not Bug Prevention
- Share
-
- |
Read later
My Reading List
In a recent post, Michael Feathers, a trainer, mentor and consultant, argues against the widely held idea that unit testing, by itself, improves code quality. Michael talks about unit testing, integration tests, TDD and Clean Room Software Development, concluding that code quality is a function of thought and reflection, not bug prevention. Steve Freeman, an independent consultant for M3P, develops Michael's idea further, talking about the "cognitive justification for TDD" and explaining why TDD is helpful.
Michael considers the idea of improving quality by catching bugs during tests as flawed:
One very common theory about unit testing is that quality comes from removing the errors that your tests catch. Superficially, this makes sense. Tests can pass or fail and when they fail we learn that we have a problem and we correct it. If you subscribe to this theory, you expect to find fewer integration errors when you do integration testing and fewer “unit” errors when you do unit testing. It’s a nice theory, but it’s wrong. The best way to see this is to compare unit testing to another way of improving quality – one that has a very dramatic measurable effect.
To prove his argument, Michael talks about Clean Room Software Development, a development method which was used during 1980s. Clean Room did not contain any unit testing according to Michael:
The notion behind Clean Room was that you could increase quality by increasing the rigor of development. In Clean Room, you had to write a logical predicate for every little piece of your code and you had to demonstrate, during a review, that your code did no more or less than the predicate described. It was a very serious approach, and it was a bit more radical than what I just described: another tenet of Clean Room was that there was to be no unit testing. None. Zilch. When you wrote your code it was assumed correct after it was reviewed. The only testing that was done was stochastic testing at the functional level.
What was interesting was that Clean Room produced results, raising the quality of the code without doing any unit testing. What happened with Clean Room and happens with TDD is similar: the developer is forced to constantly review, refactoring and improve his code. Michael's conclusion is:
My point is that we can't look at testing mechanistically. Unit testing does not improve quality just by catching errors at the unit level. And, integration testing does not improve quality just by catching errors at the integration level. The truth is more subtle than that. Quality is a function of thought and reflection - precise thought and reflection. That’s the magic. Techniques which reinforce that discipline invariably increase quality.
Starting from Michael's post, Steve Freeman, an independent consultant for M3P, develops the idea further, talking about the "cognitive justification for TDD". Developers make some decisions which impact their code:
It turns out that people don’t actually spend their time carefully working out the trade-offs and then picking the best option. Instead, we employ a “first-fit” approach: work through an ordered list of learned responses and pick the first one that looks good enough. All of this happens subconsciously, then our slower rational brain catches up and justifies the existing decision—we can’t even tell it’s happening.
The idea is to stop using the first solution that comes to our mind, and start evaluating various options, and that's why TDD is helpful according the Steve:
Test-Driven Development works (or should do) by breaking our first-fit pattern matching. It stops us being expert and steam-rolling over the problem with, literally, the first thing that came into our minds. It forces us out of our comfort zone long enough to consider the real requirements we should be addressing. Even better, starting with a test forces us to think first about the need (what’s the test for that?), and then about a solution that our expert mind is so keen to provide.
Steve gives an example to prove his point:
The best supporting evidence is Arlo Belshee’s group that implemented Promiscuous Pairing. They found empirically that they were most productive when switching pairs every couple of hours, contrary to what anyone would expect; their view was that were taking advantage of constantly being in a state of “Beginner’s Mind”.
Related Sponsor
Agile Project Management That Adapts With You. Provide your global development team a shared space that adapts to the way they work.
I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
However, I see no convincing argument from Steve Freeman that justifies his suggestion that TDD is the same. I can almost say that TDD is the opposite. The simplest thing that can possibly work? For me it is exactly the First Fit. TDD requires developers to be highly responsible and conscious not to fall in the "Simplest Thing That Can Possibly Work" trap!
Re: I am really tired of TDD being imposed in every quality discussion!
by
Jeff Santini
Re: I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
Re: I am really tired of TDD being imposed in every quality discussion!
by
Bruce Rennie
What Feathers is saying is that it's the thought process that is involved in creating the code that determines quality. TDD and Clean Room are successful, if they are, because they provide opportunity for that thought process to occur. However, if you find yourself in an environment where TDD is occurring without that reflection, then you will likely not see any benefit.
In other words, don't confuse the physical manifestation of the practice with the underlying intent. That applies to both TDD and Clean Room.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Carlos Zuniga
The simplest thing that can possibly work? For me it is exactly the First Fit. TDD requires developers to be highly responsible and conscious not to fall in the "Simplest Thing That Can Possibly Work" trap!
Even though I agree that sometimes the "Simplest Thing That Can Possibly Work" can be a trap, it can also help take the complexity in software to a minimum. Developers tend to do more than expected because "it's cool" or the "user will need it eventually" and this makes the software bloated.
I think we should try and do the "Simplest Thing That Can Possibly Work That Maintains Quality Up", and really analyze what we are doing before coding.
Just my 2 cents.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
Also I suppose that predicated of Clean Room practice are about semantic correctness, where TDD are tests with specific parameters.
In the definition of TDD you should test a small portion, and you advanced by small steps. So you never think about the integrity correctness and coherence.
TDD is NOT about integration tests. I wonder how can you write an integration test first and still call it TDD.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
I have to say though that I am ok if you do the simple thing that can possibly work if it is easily reversible. When it is not, thinking is the tool.
As James Coplien said in a prior presentation on InfoQ: If you know some stuff celebrate that you know it and put a peg in the ground.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Bruce Rennie
Feathers himself clearly views Clean Room and TDD as comparable practices. This tends to make one wonder if Clean Room could be abused in ways that Feathers fears may happen to teams that employ unit testing without understanding the importance of reflection and precise thinking about the problem. Feathers goes on to indicate his preference for TDD over Clean Room which should also indicate that the point of the article isn't about which of those two practices you employ
Re: I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
Thus what I meant by loosing focus (and I guess it even applies to a part of Feathers's post) is starting to talk about TDD benefits and loosing the analogy.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Jeff Santini
" Clean Room is more of a discipline because there is no Green bar (a technique) that makes you satisfied even if you messed up the code"
So this is an example of how a technique can be used inappropriately, which is possible with any technique based on the ideas expressed in the discussion above. By making that statement, are you suggesting that Clean room is either a) not corruptable,or b) less corruptable than TDD? Surely someone could come up with an example of how Clean Room could be used incorrectly just as your example of a successful test that "messed up the code" shows incorrect usage of TDD.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
Quality code can possibly be a result of keeping the discipline of writing quality code while doing TDD
Clean Room is a discipline about maintaining a certain code quality, period.
That is the big difference in my opinion.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Andrea Maietta
TDD means "let me think what this should do... ok I'll write a test that uses this class... there you have the class... ok it works... LET'S ELIMINATE ALL DUPLICATES AND SEE IF WE CAN SIMPLIFY IT FURTHER".
TDD helps quality not only because your software works, but also because it is simple and easy to understand (then to modify). TDD should not be imposed in every quality discussion, but it surely is a very good means to reach quality; writing tests surely is time consuming, but it is well invested time: you're paying for quality. As Philip Crosby said, quality is free but only to those who are willing to pay heavily for it. That's why also Clean Room works, because you're willing to pay.
To quote,
code quality is a function of thought and reflection, not bug prevention.
Also posted here
Re: I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
Code quality is not only about code simplification, but also and most importantly about reflecting correctly the domain (see DDD).
TDD is applied to units, you can not build a coherent quality code while thinking locally. That is why TDD is not the "Tool" to quality code.
Quality code is a result of good design skills (thorough domain analyses), a lot of discipline, and Functional tests.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Andrea Maietta
TDD is applied to units, you can not build a coherent quality code while thinking locally.
That's true, even if I'd slightly change it in "while ONLY thinking locally". But it surely helps.
Quality code is a result of good design skills (thorough domain analyses), a lot of discipline, and Functional tests.
Again, I agree with you. And, again, TDD can help, and this is particularly true if when you talk about functional tests you are referring to automated functional tests: TDD does not only refer to unit testing.
Re: I am really tired of TDD being imposed in every quality discussion!
by
Sadek Drobi
This is from www.xprogramming.com/xpmag/whatisxp.htm
"test-driven development", working in very short cycles of adding a test, then making it work.
So by definition it is about short cycles. The same is stated by Kent Back in his book. Which means think locally.
TDD can possibly produce quality code (thanks to good design skills, thorough domain analyses, a lot of discipline, and Functional tests. So no it is not about TDD) , but that is far different from saying TDD produces quality code
Re: I am really tired of TDD being imposed in every quality discussion!
by
Andrea Maietta
Re: I am really tired of TDD being imposed in every quality discussion!
by
Steve Freeman
It's only a theory that I find rings true for my experience and some other people. Arlo Belshee provides one empirical data point.
I remember the formal methods people making great claims for the benefits of using Z in the development of CICS at IBM Hursley. My (unfounded) suspicion is that the real benefit was from making very clever people slow down enough to understand what they were trying to do.
S.
Educational Content
Writing Usable APIs in Practice
Giovanni Asproni May 19, 2013
Concurrency in Clojure
Stuart Halloway May 17, 2013
Confessions of an Agile Addict
Ole Friis Østergaard May 16, 2013
Hello stranger!
You need to Register an InfoQ account or Login to post comments. But there's so much more behind being registered.Get the most out of the InfoQ experience.
Tell us what you think