BT

Return on Investment for Automated Testing

| by Amr Elssamadisy Follow 0 Followers on Apr 02, 2009. Estimated reading time: 1 minute |

Test automation is often seen as a way to reduce the costs of testing, increase test coverage and effectiveness, and shorten testing cycles. In fact many software organizations consider automation as a vital step in establishing a mature QA program and it certainly has a lot of value if it can be effectively leveraged. However, it can never eliminate the manual testing, which is crucial for thorough testing of software applications, completely.

Automated testing involves higher upfront costs and should be looked at as a long-term investment where the pay-offs come anywhere between 2-4 years down the road. One has to keep in mind that there are various intangible benefits associated with automation. Performing a return on investment (ROI) for your planned automation can however help you understand right at the beginning the actual returns that you will get from your investments and you can weigh those against the benefits you will gain from automation.

To that end, Aspire Systems has developed a Test Automation ROI Calculator with which you can determine a high level approximation of effort and cost.  The test automation calculator comes up with an approximation of effort and savings if you choose to move towards test automation based on the following factors:

Product Details

  • Type of product                   

  • Age of your product           

Technology

  • Presentation layer                   

  • Middleware                   

  • Database                   

QA Team Size

  • Manual testing team size               

  • Automation team size (if any)               

Release Details

  • Number of releases planned for the year               

Test Execution Parameters

  • Number of existing regression test cases               

  • Number of configurations to be tested               

  • Required number of regression test cycles per release               

Cost Details

  • Hourly cost per QA resource               

  • QA environment cost               

Test Automation Tool

  • Test automation tool cost               

Test Automation Parameters

  • % of test cases that can be considered for test automation               

  • Estimated base time required to build test automation suite               

  •     Less: Usage of reusable components               

  •     Add: Time required to design the automation framework            

  •     Add: Time required to build batch scripts            

Cost Details

  • Hourly cost per test automation resource              

Automation is a strategic decision and an understanding on the potential ROI will help in the decision to invest in automation. This tool may be help you determine what works for you.  

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Ha ha very funny by Aslak Hellesøy

But April fools was 2 days ago...

I filled out the form, and was unable to enter anything in field 5 from bottom. Hit calculate and got no result, no error message and all my data was gone. If this thing was tested, it was done poorly.

Nothing to see here. Move on.

Re: Ha ha very funny by Amr Elssamadisy

:) Try something other than firefox. Good call. It works with Safari on my mac but no with firefox.

Estimated time/cost for maintenance? Estimated gain from reduced cyce time? by Eric Schumacher

I might have missed it, but where is the number for estimated time for maintenance and updates? The long term care and feeding is costly as well.

Where are the bits that estimate the dollar gain from reduced cycle time relative to shortening delivery/increase in quality?

Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t by Sai Venkatakrishnan

I am not able to believe this. On one side infoQ provides news about Lisa Crispin's Agile Testing book and on the other side we have something like this which totally contradicts that view.

I have tried all this ROI stuff and even was involved in developing one before and there are several things which are clear from them.
1) They make an assumption that testing can be done after development. Ir encourages test last attitude
2) You can't do automation without manual regression. Automation is about automating manual test cases.
3) I never advocate them as I don't think things are so deterministic.

These stunts are used by offshore test vendors to attract clients (I had worked with such a vendor before so this comes out of experience). Please read this beautiful post by Elisabeth Hendrickson about how Agile changes the equations in Automation. testobsessed.com/2009/02/10/how-much-to-automat....

I hope infoQ helps people understand ways to build better software rather than post every news which comes to its way.

Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t by Amr Elssamadisy


I am not able to believe this. On one side infoQ provides news about Lisa Crispin's Agile Testing book and on the other side we have something like this which totally contradicts that view.


First of all, InfoQ does not recommend this tool, nor does it recommend against it. It is news - information that we thought our readers would find useful.

Secondly, from this newswriter's point of view, there is no black or white, everything depends on context. If you have a problem with the tool, by all means, don't use it. On the other hand, if you are considering recommending automating a large percentage of manual tests on your current legacy project, it may be useful.

As for me, I found the tool intriguing and haven't really spent enough time on it to understand the assumptions it makes. Nonetheless, automating existing tests is a recommendation I make to most teams I work with (not all, because that depends on context). Talking to executives in their language, ROI, is always a plus.

Finally, automating existing tests written by a QA department is independent of TDD. This assumes a QA department, which, by the way, many teams have and even when they go to cross-functional-teams, will still have testers.

100% automation coverage by Michel Löhr

I my current project I am responsible for QA (also process). And I have created a small testframework based on Groovy which uses a (german) DSL to test the Java Swing based application. Before my arrival there were no automated tests but now every developer writes their own GUI(unit)test which we put in our regressiontest in Hudson. This DSL allows for a testcase to almost look alike a testcase design for a manual testcase which is very productive to use.

Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t by Narayanan Moorthy

I am not able to believe this. On one side infoQ provides news about Lisa Crispin's Agile Testing book and on the other side we have something like this which totally contradicts that view.

I have tried all this ROI stuff and even was involved in developing one before and there are several things which are clear from them.
1) They make an assumption that testing can be done after development. Ir encourages test last attitude
2) You can't do automation without manual regression. Automation is about automating manual test cases.
3) I never advocate them as I don't think things are so deterministic.

These stunts are used by offshore test vendors to attract clients (I had worked with such a vendor before so this comes out of experience). Please read this beautiful post by Elisabeth Hendrickson about how Agile changes the equations in Automation. testobsessed.com/2009/02/10/how-much-to-automat....

I hope infoQ helps people understand ways to build better software rather than post every news which comes to its way.

Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t by Narayanan Moorthy

I am not sure how you are just saying this ROI calculator is a stunt. This might be developed to for the purpose of serving the testing community. I think it tries to bring more awareness and i dont think there are ROI calculator available on the web similar to this. If you could generate the report from their site, u can see the value. Try once again.

Re: Ha ha very funny by Narayanan Moorthy

Now this works in all the browsers viz., firefox, IE, Safari and Chrome. You can use this ROI calc to find the ROI.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

9 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT