Automated testing is all the rage, but is it everything? Micahel, a Test Technical Lead at Microsoft, asks "How do you know whether you have automated enough - or too much?"
The case for automated testing is easy. For a bit of effort up front, the code can be tested for regressions on a regular with little or no developer involvement. However like most techniques, it does not always work out as planned.
Automated tests are by nature scripted, not exploratory. Even with an automation stack which injects all sorts of variability, the tests wear grooves in those areas of the product they cover and they ignore everything else. When something unexpected happens they are likely to die, and even if they are able to recover they are not able to stop what they were doing and investigate that unexpected happening. And don't forget the maintenance required to keep those tests running - which efforts are not helping you find defects in your application. Say, have you had time to actually use your application yet?
Micahel goes on to discuss the advantages and disadvantages of manual testing including the coverage that comes from exploratory testing and inability to do a full test after every build.
On the other extreme is the Automate Nothing approach. Here every test case is executed manually by a person physically using their mouse and keyboard. This has considerable payoffs: every test can be exploratory. The entire surface of the product will likely be covered. When something unexpected happens it is easily followed up. No maintenance is required to keep the test cases up to date with changes in the application. Everybody is always using the application. Pretty much nirvana, right?
He ends with a question, "It is clear to me that Automating Everything is taking things too far. So is Automating Nothing. I have not yet found a balance I like. How about you?".
Community comments
Not this again...
by Bruce Rennie,
Re: Not this again...
by Jonathan Allen,
Re: Not this again...
by Bruce Rennie,
Governance and test automation
by Frank Cohen,
Not this again...
by Bruce Rennie,
Your message is awaiting moderation. Thank you for participating in the discussion.
I've seen this argument drag on in other agile forums for quite some time now and, I admit, I truly don't get it.
This feels like an attempt to create controversy where none should exist. I guess part of it can be attributed to the strong stand of agile developers such as Ron Jeffries who press hard for the "automate everything" case but I think the proponents of exploratory testing misinterpret that stand. Part of the problem is that there feels like an underlying assumption in some that you can do it all. For many of us, that's never going to happen. Instead, we have to figure out where the biggest ROI for testing lies and go with that.
If you believe that exploratory testing provides the biggest bang for your testing buck, then you should do it. Simple as that. If you feel that TDD and an automated regression suite provides more value, you should do that. Worrying about what proportion should be manual or not seems besides the point.
Personally, I can see the value of exploratory testing so I place myself in the "automate everything you can so you actually have time to do those things you can't automate" camp.
Governance and test automation
by Frank Cohen,
Your message is awaiting moderation. Thank you for participating in the discussion.
Micahel's blog entry is a conversation starter.
Seems to me that he is missing "governance" when talking about test automation. In the SOA context governance adds a registry to announce that a servivce eixsts and a set of policies to control its use. Imagine a policy that says 'You must run a test script to check that this service is correctly configured before using this service." The governance tool manages the policies, stores the functional tests scripts, issues a call to run the test, and saves the results. Take a look at governance tools (BEA ALER, WebMethods X-Registry, Iona Repository, etc.) to get a better idea on how to mitigate the "test everywhere" mentality into a "test by policy" method.
-Frank Cohen
www.pushtotest.com
Re: Not this again...
by Jonathan Allen,
Your message is awaiting moderation. Thank you for participating in the discussion.
That was not my intention for running this piece. I'm looking more at the "Hey, why aren't we talking about anything besides automated testing?" angle. Lately out industury, and thus InfoQ, has focused on unit testing frameworks to the exclusion of everything else.
What I would like to see come from this is not a debate on whether or not to automate tests, but rather more discussion on what else it out there.
Re: Not this again...
by Bruce Rennie,
Your message is awaiting moderation. Thank you for participating in the discussion.
I think we talk about automated testing because:
1. It gets developers involved
2. It's better than what 90% of most software shops were doing previously anyway
3. In most situations, it has a pretty decent ROI
4. With TDD, it's pro-active rather than reactive.
Seriously, what's not to like? :)
I think that part of the problem is that the question is posed as an "either/or" situation. To me, it's not. Automated tests are the tool I use to create more time to do exploratory testing. No automated tests means I'm probably not doing exploratory testing anyway because my staff are too busy manually testing the daily work products.
That said, I've worked with agile teams for 5+ years now and I've never once worked with a team that wasn't constantly questioning the value of automating any given test. We may say "write a test for everything" but everyone always knew enough to watch for diminishing returns. Sort of the same way that refactor mercilessly doesn't mean refactor suicidally.