In a recent post, Michael Feathers, a trainer, mentor and consultant, argues against the widely held idea that unit testing, by itself, improves code quality. Michael talks about unit testing, integration tests, TDD and Clean Room Software Development, concluding that code quality is a function of thought and reflection, not bug prevention. Steve Freeman, an independent consultant for M3P, develops Michael's idea further, talking about the "cognitive justification for TDD" and explaining why TDD is helpful.
Michael considers the idea of improving quality by catching bugs during tests as flawed:
One very common theory about unit testing is that quality comes from removing the errors that your tests catch. Superficially, this makes sense. Tests can pass or fail and when they fail we learn that we have a problem and we correct it. If you subscribe to this theory, you expect to find fewer integration errors when you do integration testing and fewer “unit” errors when you do unit testing. It’s a nice theory, but it’s wrong. The best way to see this is to compare unit testing to another way of improving quality – one that has a very dramatic measurable effect.
To prove his argument, Michael talks about Clean Room Software Development, a development method which was used during 1980s. Clean Room did not contain any unit testing according to Michael:
The notion behind Clean Room was that you could increase quality by increasing the rigor of development. In Clean Room, you had to write a logical predicate for every little piece of your code and you had to demonstrate, during a review, that your code did no more or less than the predicate described. It was a very serious approach, and it was a bit more radical than what I just described: another tenet of Clean Room was that there was to be no unit testing. None. Zilch. When you wrote your code it was assumed correct after it was reviewed. The only testing that was done was stochastic testing at the functional level.
What was interesting was that Clean Room produced results, raising the quality of the code without doing any unit testing. What happened with Clean Room and happens with TDD is similar: the developer is forced to constantly review, refactoring and improve his code. Michael's conclusion is:
My point is that we can't look at testing mechanistically. Unit testing does not improve quality just by catching errors at the unit level. And, integration testing does not improve quality just by catching errors at the integration level. The truth is more subtle than that. Quality is a function of thought and reflection - precise thought and reflection. That’s the magic. Techniques which reinforce that discipline invariably increase quality.
Starting from Michael's post, Steve Freeman, an independent consultant for M3P, develops the idea further, talking about the "cognitive justification for TDD". Developers make some decisions which impact their code:
It turns out that people don’t actually spend their time carefully working out the trade-offs and then picking the best option. Instead, we employ a “first-fit” approach: work through an ordered list of learned responses and pick the first one that looks good enough. All of this happens subconsciously, then our slower rational brain catches up and justifies the existing decision—we can’t even tell it’s happening.
The idea is to stop using the first solution that comes to our mind, and start evaluating various options, and that's why TDD is helpful according the Steve:
Test-Driven Development works (or should do) by breaking our first-fit pattern matching. It stops us being expert and steam-rolling over the problem with, literally, the first thing that came into our minds. It forces us out of our comfort zone long enough to consider the real requirements we should be addressing. Even better, starting with a test forces us to think first about the need (what’s the test for that?), and then about a solution that our expert mind is so keen to provide.
Steve gives an example to prove his point:
The best supporting evidence is Arlo Belshee’s group that implemented Promiscuous Pairing. They found empirically that they were most productive when switching pairs every couple of hours, contrary to what anyone would expect; their view was that were taking advantage of constantly being in a state of “Beginner’s Mind”.
Community comments
I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Jeff Santini,
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Bruce Rennie,
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Bruce Rennie,
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Jeff Santini,
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Carlos Zuniga,
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Andrea Maietta,
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Andrea Maietta,
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Re: I am really tired of TDD being imposed in every quality discussion!
by Andrea Maietta,
Re: I am really tired of TDD being imposed in every quality discussion!
by Steve Freeman,
I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
Excellent post by Feather. I guess a Clean Room is an interesting approach that can produce really good code quality and can improve developer's skills.
However, I see no convincing argument from Steve Freeman that justifies his suggestion that TDD is the same. I can almost say that TDD is the opposite. The simplest thing that can possibly work? For me it is exactly the First Fit. TDD requires developers to be highly responsible and conscious not to fall in the "Simplest Thing That Can Possibly Work" trap!
Re: I am really tired of TDD being imposed in every quality discussion!
by Jeff Santini,
Your message is awaiting moderation. Thank you for participating in the discussion.
I am not aware of any imposition that TDD must be part of any quality discussion. Why don't you start a quality discussion without any TDD? That way we could test your theory.
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
@Jeff: excuse me, but I don't understand which theory you are referring to. I just do not understand why should I hear each time about "TDD is good for quality code" when the whole point is about discovering a brilliant practice like Clean Room. It often breaks the focus of the article.
Re: I am really tired of TDD being imposed in every quality discussion!
by Bruce Rennie,
Your message is awaiting moderation. Thank you for participating in the discussion.
To focus on "Clean Room" is to miss the point, I think.
What Feathers is saying is that it's the thought process that is involved in creating the code that determines quality. TDD and Clean Room are successful, if they are, because they provide opportunity for that thought process to occur. However, if you find yourself in an environment where TDD is occurring without that reflection, then you will likely not see any benefit.
In other words, don't confuse the physical manifestation of the practice with the underlying intent. That applies to both TDD and Clean Room.
Re: I am really tired of TDD being imposed in every quality discussion!
by Carlos Zuniga,
Your message is awaiting moderation. Thank you for participating in the discussion.
Even though I agree that sometimes the "Simplest Thing That Can Possibly Work" can be a trap, it can also help take the complexity in software to a minimum. Developers tend to do more than expected because "it's cool" or the "user will need it eventually" and this makes the software bloated.
I think we should try and do the "Simplest Thing That Can Possibly Work That Maintains Quality Up", and really analyze what we are doing before coding.
Just my 2 cents.
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
In Clean Room there is a review time. So it is more of a group work unlike TDD (I know you will tell me about pair programming but that is another story)
Also I suppose that predicated of Clean Room practice are about semantic correctness, where TDD are tests with specific parameters.
In the definition of TDD you should test a small portion, and you advanced by small steps. So you never think about the integrity correctness and coherence.
TDD is NOT about integration tests. I wonder how can you write an integration test first and still call it TDD.
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
@Carlos: That is exactly the problem, quality in "That Maintains Quality UP" is subjective and it can produce a lot of interpretations. I don't guess we lack golden rules as much as we lack good software designers that at least understand what abstraction is.
I have to say though that I am ok if you do the simple thing that can possibly work if it is easily reversible. When it is not, thinking is the tool.
As James Coplien said in a prior presentation on InfoQ: If you know some stuff celebrate that you know it and put a peg in the ground.
Re: I am really tired of TDD being imposed in every quality discussion!
by Bruce Rennie,
Your message is awaiting moderation. Thank you for participating in the discussion.
Again, with all due respect, I think you're missing the point. The article Feathers wrote isn't about introducing some new practice called Clean Room. It's about relying on practices to improve quality without understanding the fundamental component of the practice that makes that possible.
Feathers himself clearly views Clean Room and TDD as comparable practices. This tends to make one wonder if Clean Room could be abused in ways that Feathers fears may happen to teams that employ unit testing without understanding the importance of reflection and precise thinking about the problem. Feathers goes on to indicate his preference for TDD over Clean Room which should also indicate that the point of the article isn't about which of those two practices you employ
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
@Bruce: You are right, Feathers is not comparing TDD and Clean Room. But what I find interesting in his post is the emphasizes on discipline over techniques. TDD is a technique, and I guess it is missing the point when you start mixing up both. Clean Room is more of a discipline because there is no Green bar (a technique) that makes you satisfied even if you messed up the code.
Thus what I meant by loosing focus (and I guess it even applies to a part of Feathers's post) is starting to talk about TDD benefits and loosing the analogy.
Re: I am really tired of TDD being imposed in every quality discussion!
by Jeff Santini,
Your message is awaiting moderation. Thank you for participating in the discussion.
@Sadek
" Clean Room is more of a discipline because there is no Green bar (a technique) that makes you satisfied even if you messed up the code"
So this is an example of how a technique can be used inappropriately, which is possible with any technique based on the ideas expressed in the discussion above. By making that statement, are you suggesting that Clean room is either a) not corruptable,or b) less corruptable than TDD? Surely someone could come up with an example of how Clean Room could be used incorrectly just as your example of a successful test that "messed up the code" shows incorrect usage of TDD.
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
@Jeff
Quality code can possibly be a result of keeping the discipline of writing quality code while doing TDD
Clean Room is a discipline about maintaining a certain code quality, period.
That is the big difference in my opinion.
Re: I am really tired of TDD being imposed in every quality discussion!
by Andrea Maietta,
Your message is awaiting moderation. Thank you for participating in the discussion.
When you have a "simplest thing that could possibly work" the first fit can be a good answer, but it could also be a horrible one. That is where refactoring comes in, and I must admit I'm pretty astounded that nobody mentioned it so far.
TDD means "let me think what this should do... ok I'll write a test that uses this class... there you have the class... ok it works... LET'S ELIMINATE ALL DUPLICATES AND SEE IF WE CAN SIMPLIFY IT FURTHER".
TDD helps quality not only because your software works, but also because it is simple and easy to understand (then to modify). TDD should not be imposed in every quality discussion, but it surely is a very good means to reach quality; writing tests surely is time consuming, but it is well invested time: you're paying for quality. As Philip Crosby said, quality is free but only to those who are willing to pay heavily for it. That's why also Clean Room works, because you're willing to pay.
To quote,
Also posted here
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
@Andrea
Code quality is not only about code simplification, but also and most importantly about reflecting correctly the domain (see DDD).
TDD is applied to units, you can not build a coherent quality code while thinking locally. That is why TDD is not the "Tool" to quality code.
Quality code is a result of good design skills (thorough domain analyses), a lot of discipline, and Functional tests.
Re: I am really tired of TDD being imposed in every quality discussion!
by Andrea Maietta,
Your message is awaiting moderation. Thank you for participating in the discussion.
That's true, even if I'd slightly change it in "while ONLY thinking locally". But it surely helps.
Again, I agree with you. And, again, TDD can help, and this is particularly true if when you talk about functional tests you are referring to automated functional tests: TDD does not only refer to unit testing.
Re: I am really tired of TDD being imposed in every quality discussion!
by Sadek Drobi,
Your message is awaiting moderation. Thank you for participating in the discussion.
@Andrea:
This is from www.xprogramming.com/xpmag/whatisxp.htm
So by definition it is about short cycles. The same is stated by Kent Back in his book. Which means think locally.
TDD can possibly produce quality code (thanks to good design skills, thorough domain analyses, a lot of discipline, and Functional tests. So no it is not about TDD) , but that is far different from saying TDD produces quality code
Re: I am really tired of TDD being imposed in every quality discussion!
by Andrea Maietta,
Your message is awaiting moderation. Thank you for participating in the discussion.
Again, I agree with you. I am only saying that TDD HELPS to produce good gode, but, to use your own words, is not "the" tool. I mean, if you have a perfectly analyzed class which perfectly represents a domain object (I'm fond on DDD too) but you find it difficult to be tested, probably there is a slightly different and slightly better metaphor to describe it(and maybe an SME can help in defining it). Being able to write a test for a class before actually writing it kind of "forces" you to have an attitute just like the one you describe: you have to analyze the domain with a lot of discipline, because with TDD you actually are saying "what am I expecting from this piece of code? what is the expected behavior of the real object it represents?". Of course, you must also have funcional tests, and having good design skills is almost mandatory otherwise one would blatantly miss the point of the whole question.
Re: I am really tired of TDD being imposed in every quality discussion!
by Steve Freeman,
Your message is awaiting moderation. Thank you for participating in the discussion.
My theory about the cognitive effects of TDD was that it helps me concentrate on what I really need to the code to do. It helps me avoid jumping in with a solution before I understand the problem. TDD makes no guarantees about what will if I don't have the skills to respond to that need, or I don't fulfil the second half of the deal by cleaning up as I go. Sorry about that.
It's only a theory that I find rings true for my experience and some other people. Arlo Belshee provides one empirical data point.
I remember the formal methods people making great claims for the benefits of using Z in the development of CICS at IBM Hursley. My (unfounded) suspicion is that the real benefit was from making very clever people slow down enough to understand what they were trying to do.
S.