Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Testing: Manual or Automated?

Testing: Manual or Automated?

Automated testing is all the rage, but is it everything? Micahel, a Test Technical Lead at Microsoft, asks "How do you know whether you have automated enough - or too much?"

The case for automated testing is easy. For a bit of effort up front, the code can be tested for regressions on a regular with little or no developer involvement. However like most techniques, it does not always work out as planned. 

Automated tests are by nature scripted, not exploratory. Even with an automation stack which injects all sorts of variability, the tests wear grooves in those areas of the product they cover and they ignore everything else. When something unexpected happens they are likely to die, and even if they are able to recover they are not able to stop what they were doing and investigate that unexpected happening. And don't forget the maintenance required to keep those tests running - which efforts are not helping you find defects in your application. Say, have you had time to actually use your application yet?

Micahel goes on to discuss the advantages and disadvantages of manual testing including the coverage that comes from exploratory testing and inability to do a full test after every build.

On the other extreme is the Automate Nothing approach. Here every test case is executed manually by a person physically using their mouse and keyboard. This has considerable payoffs: every test can be exploratory. The entire surface of the product will likely be covered. When something unexpected happens it is easily followed up. No maintenance is required to keep the test cases up to date with changes in the application. Everybody is always using the application. Pretty much nirvana, right?

He ends with a question, "It is clear to me that Automating Everything is taking things too far. So is Automating Nothing. I have not yet found a balance I like. How about you?".

Rate this Article