Testing: Manual or Automated?

| by Jonathan Allen Follow 614 Followers on Apr 16, 2007. Estimated reading time: 1 minute |

Automated testing is all the rage, but is it everything? Micahel, a Test Technical Lead at Microsoft, asks "How do you know whether you have automated enough - or too much?"

The case for automated testing is easy. For a bit of effort up front, the code can be tested for regressions on a regular with little or no developer involvement. However like most techniques, it does not always work out as planned. 

Automated tests are by nature scripted, not exploratory. Even with an automation stack which injects all sorts of variability, the tests wear grooves in those areas of the product they cover and they ignore everything else. When something unexpected happens they are likely to die, and even if they are able to recover they are not able to stop what they were doing and investigate that unexpected happening. And don't forget the maintenance required to keep those tests running - which efforts are not helping you find defects in your application. Say, have you had time to actually use your application yet?

Micahel goes on to discuss the advantages and disadvantages of manual testing including the coverage that comes from exploratory testing and inability to do a full test after every build.

On the other extreme is the Automate Nothing approach. Here every test case is executed manually by a person physically using their mouse and keyboard. This has considerable payoffs: every test can be exploratory. The entire surface of the product will likely be covered. When something unexpected happens it is easily followed up. No maintenance is required to keep the test cases up to date with changes in the application. Everybody is always using the application. Pretty much nirvana, right?

He ends with a question, "It is clear to me that Automating Everything is taking things too far. So is Automating Nothing. I have not yet found a balance I like. How about you?".

Rate this Article

Adoption Stage

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Not this again... by Bruce Rennie

I've seen this argument drag on in other agile forums for quite some time now and, I admit, I truly don't get it.

This feels like an attempt to create controversy where none should exist. I guess part of it can be attributed to the strong stand of agile developers such as Ron Jeffries who press hard for the "automate everything" case but I think the proponents of exploratory testing misinterpret that stand. Part of the problem is that there feels like an underlying assumption in some that you can do it all. For many of us, that's never going to happen. Instead, we have to figure out where the biggest ROI for testing lies and go with that.

If you believe that exploratory testing provides the biggest bang for your testing buck, then you should do it. Simple as that. If you feel that TDD and an automated regression suite provides more value, you should do that. Worrying about what proportion should be manual or not seems besides the point.

Personally, I can see the value of exploratory testing so I place myself in the "automate everything you can so you actually have time to do those things you can't automate" camp.

Governance and test automation by Frank Cohen

Micahel's blog entry is a conversation starter.

Seems to me that he is missing "governance" when talking about test automation. In the SOA context governance adds a registry to announce that a servivce eixsts and a set of policies to control its use. Imagine a policy that says 'You must run a test script to check that this service is correctly configured before using this service." The governance tool manages the policies, stores the functional tests scripts, issues a call to run the test, and saves the results. Take a look at governance tools (BEA ALER, WebMethods X-Registry, Iona Repository, etc.) to get a better idea on how to mitigate the "test everywhere" mentality into a "test by policy" method.

-Frank Cohen

Re: Not this again... by Jonathan Allen

This feels like an attempt to create controversy where none should exist.

That was not my intention for running this piece. I'm looking more at the "Hey, why aren't we talking about anything besides automated testing?" angle. Lately out industury, and thus InfoQ, has focused on unit testing frameworks to the exclusion of everything else.

What I would like to see come from this is not a debate on whether or not to automate tests, but rather more discussion on what else it out there.

Re: Not this again... by Bruce Rennie

I think we talk about automated testing because:

1. It gets developers involved
2. It's better than what 90% of most software shops were doing previously anyway
3. In most situations, it has a pretty decent ROI
4. With TDD, it's pro-active rather than reactive.

Seriously, what's not to like? :)

I think that part of the problem is that the question is posed as an "either/or" situation. To me, it's not. Automated tests are the tool I use to create more time to do exploratory testing. No automated tests means I'm probably not doing exploratory testing anyway because my staff are too busy manually testing the daily work products.

That said, I've worked with agile teams for 5+ years now and I've never once worked with a team that wasn't constantly questioning the value of automating any given test. We may say "write a test for everything" but everyone always knew enough to watch for diminishing returns. Sort of the same way that refactor mercilessly doesn't mean refactor suicidally.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

4 Discuss

Login to InfoQ to interact with what matters most to you.

Recover your password...


Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.


More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.


Stay up-to-date

Set up your notifications and don't miss out on content that matters to you