Return on Investment for Automated Testing

Posted by Amr Elssamadisy on Apr 02, 2009

Sections

Process & Practices

Topics

Agile ,

Agile in the Enterprise

Tags

Testing ,

Automation

Share |

Test automation is often seen as a way to reduce the costs of testing, increase test coverage and effectiveness, and shorten testing cycles. In fact many software organizations consider automation as a vital step in establishing a mature QA program and it certainly has a lot of value if it can be effectively leveraged. However, it can never eliminate the manual testing, which is crucial for thorough testing of software applications, completely.

Automated testing involves higher upfront costs and should be looked at as a long-term investment where the pay-offs come anywhere between 2-4 years down the road. One has to keep in mind that there are various intangible benefits associated with automation. Performing a return on investment (ROI) for your planned automation can however help you understand right at the beginning the actual returns that you will get from your investments and you can weigh those against the benefits you will gain from automation.

To that end, Aspire Systems has developed a Test Automation ROI Calculator with which you can determine a high level approximation of effort and cost.  The test automation calculator comes up with an approximation of effort and savings if you choose to move towards test automation based on the following factors:

Product Details

• Type of product                   

• Age of your product           

Technology

• Presentation layer                   

• Middleware                   

• Database                   

QA Team Size

• Manual testing team size               

• Automation team size (if any)               

Release Details

• Number of releases planned for the year               

Test Execution Parameters

• Number of existing regression test cases               

• Number of configurations to be tested               

• Required number of regression test cycles per release               

Cost Details

• Hourly cost per QA resource               

• QA environment cost               

Test Automation Tool

• Test automation tool cost               

Test Automation Parameters

• % of test cases that can be considered for test automation               

• Estimated base time required to build test automation suite               

•     Less: Usage of reusable components               

•     Add: Time required to design the automation framework            

•     Add: Time required to build batch scripts            

Cost Details

• Hourly cost per test automation resource              

Automation is a strategic decision and an understanding on the potential ROI will help in the decision to invest in automation. This tool may be help you determine what works for you.  

RelatedVendorContent

18 agile and lean practices for effective software development governance

Five Key Practices to Agile ALM

Agility at scale, become as agile as you can be

A Guide to Branching and Merging Patterns

The Agile Tester

Related Sponsor

In today’s hyper-competitive world, later may be too late to adopt Agile development and this Roadmap for Success will help you get started. Download "Agile Development: A Manager's Roadmap for Success" now!

1. Back to top

   Ha ha very funny

   by Aslak Hellesøy

   But April fools was 2 days ago...
   
   I filled out the form, and was unable to enter anything in field 5 from bottom. Hit calculate and got no result, no error message and all my data was gone. If this thing was tested, it was done poorly.
   
   Nothing to see here. Move on.

   Reply

2. Back to top

   Re: Ha ha very funny

   by Amr Elssamadisy

   :) Try something other than firefox. Good call. It works with Safari on my mac but no with firefox.

   Reply

3. Back to top

   Estimated time/cost for maintenance? Estimated gain from reduced cyce time?

   by Eric Schumacher

   I might have missed it, but where is the number for estimated time for maintenance and updates? The long term care and feeding is costly as well.
   
   Where are the bits that estimate the dollar gain from reduced cycle time relative to shortening delivery/increase in quality?

   Reply

4. Back to top

   Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t

   by Sai Venkatakrishnan

   I am not able to believe this. On one side infoQ provides news about Lisa Crispin's Agile Testing book and on the other side we have something like this which totally contradicts that view.
   
   I have tried all this ROI stuff and even was involved in developing one before and there are several things which are clear from them.
   1) They make an assumption that testing can be done after development. Ir encourages test last attitude
   2) You can't do automation without manual regression. Automation is about automating manual test cases.
   3) I never advocate them as I don't think things are so deterministic.
   
   These stunts are used by offshore test vendors to attract clients (I had worked with such a vendor before so this comes out of experience). Please read this beautiful post by Elisabeth Hendrickson about how Agile changes the equations in Automation. testobsessed.com/2009/02/10/how-much-to-automat....
   
   I hope infoQ helps people understand ways to build better software rather than post every news which comes to its way.

   Reply

5. Back to top

   Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t

   by Amr Elssamadisy

   
   I am not able to believe this. On one side infoQ provides news about Lisa Crispin's Agile Testing book and on the other side we have something like this which totally contradicts that view.
   

   
   
   First of all, InfoQ does not recommend this tool, nor does it recommend against it. It is news - information that we thought our readers would find useful.
   
   Secondly, from this newswriter's point of view, there is no black or white, everything depends on context. If you have a problem with the tool, by all means, don't use it. On the other hand, if you are considering recommending automating a large percentage of manual tests on your current legacy project, it may be useful.
   
   As for me, I found the tool intriguing and haven't really spent enough time on it to understand the assumptions it makes. Nonetheless, automating existing tests is a recommendation I make to most teams I work with (not all, because that depends on context). Talking to executives in their language, ROI, is always a plus.
   
   Finally, automating existing tests written by a QA department is independent of TDD. This assumes a QA department, which, by the way, many teams have and even when they go to cross-functional-teams, will still have testers.

   Reply

6. Back to top

   100% automation coverage

   by Michel Löhr

   I my current project I am responsible for QA (also process). And I have created a small testframework based on Groovy which uses a (german) DSL to test the Java Swing based application. Before my arrival there were no automated tests but now every developer writes their own GUI(unit)test which we put in our regressiontest in Hudson. This DSL allows for a testcase to almost look alike a testcase design for a manual testcase which is very productive to use.

   Reply

7. Back to top

   Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t

   by Narayanan Moorthy

   I am not able to believe this. On one side infoQ provides news about Lisa Crispin's Agile Testing book and on the other side we have something like this which totally contradicts that view.
   
   I have tried all this ROI stuff and even was involved in developing one before and there are several things which are clear from them.
   1) They make an assumption that testing can be done after development. Ir encourages test last attitude
   2) You can't do automation without manual regression. Automation is about automating manual test cases.
   3) I never advocate them as I don't think things are so deterministic.
   
   These stunts are used by offshore test vendors to attract clients (I had worked with such a vendor before so this comes out of experience). Please read this beautiful post by Elisabeth Hendrickson about how Agile changes the equations in Automation. testobsessed.com/2009/02/10/how-much-to-automat....
   
   I hope infoQ helps people understand ways to build better software rather than post every news which comes to its way.

   Reply

8. Back to top

   Re: Estimated time/cost for maintenance? Estimated gain from reduced cyce t

   by Narayanan Moorthy

   I am not sure how you are just saying this ROI calculator is a stunt. This might be developed to for the purpose of serving the testing community. I think it tries to bring more awareness and i dont think there are ROI calculator available on the web similar to this. If you could generate the report from their site, u can see the value. Try once again.

   Reply

9. Back to top

   Re: Ha ha very funny

   by Narayanan Moorthy

   Now this works in all the browsers viz., firefox, IE, Safari and Chrome. You can use this ROI calc to find the ROI.

   Reply

• Older >