With the presence of high quality test generation tools like Agitar One and Parasoft's JTest, some are questioning the need to write tests manually. Uncle Bob (Martin) weighted in, exploring the weakness of the idea.
These tools are designed to examine an existing body of code. Based on branches, loops etc. they synthesize a set of observations about the code. The developer reviews the observations and chooses the relevant ones to create a set of test cases. In exploring legacy code this can be a very powerful way of understanding the existing behavior and creating a safety net before making changes.
However like any technology there are limits: these tools are only generating a set of observations about the code. They don't understand algorithms or the developer's intent. From Bob:
As a simple example of this, I have tried to generate tests for the bowling game program using two of the better known test generation tools. The interface to the Bowling Game looks like this:
public class BowlingGame {The idea is that you call roll each time the balls gets rolled, and you call score at the end of the game to get the score for that game.
public void roll(int pins) {...}
public int score() {...}
}The test generators could not randomly generate valid games. It's not hard to see why. A valid game is a sequence of between 12 and 21 rolls, all of which must be integers between 0 and 10. What's more, within a given frame, the sum of rolls cannot exceed 10. These constraints are just too tight for a random generator to achieve within the current age of the universe.
On the other hand, Test Driven Development (TDD), the process of writing the tests before writing the production code, works very differently. TDD works because it provides immediate feedback about the code the developer writes. Make a small change, run the test and the developer knows if the change was good. TDD ensures that the code matches our stated intentions (the tests). TDD makes us think about the design of the code from the point of view the consumer and not just the author. It makes us think about every conditional and loop. Additionally, as James Carr mentions, TDD forces us to consider how tightly coupled our code is. From Bob again:
Using a test generator breaks this concept because the generator writes the test using the production code as input. The generated test is not a human restatement, it is an automatic translation. The human states intent only once, and therefore does not gain insights from restatement, nor does the generated test check that the intent of the code was achieved. It is true that the human must verify the observations, but compared to TDD that is a far more passive action, providing far less insight into defects, design and intent.
... This does not mean that test generators aren't useful. ... I think they can help to partially characterize a large base of legacy code.
Community comments
I think Agitar (JUnit Factory) is pretty honest
by Porter Woodward,
Re: I think Agitar (JUnit Factory) is pretty honest
by Mark Levison,
I think Agitar (JUnit Factory) is pretty honest
by Porter Woodward,
Your message is awaiting moderation. Thank you for participating in the discussion.
On JUnit factory they are very honest about what their tool generates: Characterization Tests. Artima had a pretty good series of articles about them. There really is no way for the test to generate code for the given example of a bowling game... Because the game logic/rules haven't been encoded as code. So there's no validation as to what is a valid game or not. That might say more about a failing of the code example than the test generators. TDD and Agile tend to say the code is the documentation; but if you're not expressing rules and validations in the code, where is it expressed? And how can any tool take advantage of it?
Re: I think Agitar (JUnit Factory) is pretty honest
by Mark Levison,
Your message is awaiting moderation. Thank you for participating in the discussion.
I agree Agitar is entirely honest, I don't know Parasoft well enough to comment. Their CTO Alberto Savoia has made statements on several occasions that Agitar's tools don't replace TDD.
You're also right that the tool can't encode the rules because it hasn't been explicitly encoded in the Bowling Game. BTW I've not seen Bob's bowling code so that's only an assumption on my part.
However even if the test generator did a perfect job of understanding the rules of bowling I still don't think it would capture the essence of TDD:
- forces us to write small decoupled chunks of code
- forces us to simplify
- causes to consider the code from the context of its user as well as the code itself.
By definition Test Generators miss this benefit because they come into play after you've written the code.
I think that Agitar's tool is very useful but it doesn't replace TDD.
I will try to get Alberto's attention to comment on this story.