BT

Developing Test Automation Scripts and Automation Frameworks

Posted by Sasha Reminnyi on Oct 24, 2015 |

Frequently during the early stages of test automation project consulting I am asked for estimates around automation implementation. And when I first give the numbers the next question – “is that for a test suite or for a framework?”

This question always makes me a bit stuck – I don’t know what I’m being asked for… What are test scripts? What is the framework? Can they be separated?

Let’s start with some definitions.

Automation tools

Automation tools/instruments are used to interact with the UI – emulate button clicks, type text, verify text box values. At least here I can say for sure – there is no ambiguity.

Frameworks

Some time ago my vision about a framework was rather specific –reusable, SUT (system under test) agnostic and automation tool agnostic libraries to speed up automation implementation.

But in the IT world the word framework is so frequently used, that even when speaking about automated testing only it has different meaning depending on the context.

Pic 1 –different frameworks examples

Tool specific frameworks

Commercial automation tool vendors or open source communities frequently develop a whole infrastructure around their tools that allow generating reports, running test suites, distributed test execution, working with test execution environments. For instance, the Selenium tool’s main component is WebDriver – which works like a plugin for a web browser to manipulate the DOM model (xml-based document object model of the web page) of the web application from inside the web browser. But there are also a number of additional coding libraries, and a record-playback tool (Selenuium IDE). All of those together form the Selenium test automation framework.

Another great example would be Serenity that is also tool specific (built around Selenium Web Driver) framework. But its goal is not to have many optional components/plugins. It can be considered as an accelerator – specific components are already combined by the community experts into one substance - to support the possibility of faster test automation implementation startup and also support for BDD approaches.

In the commercial world, it’s even harder to identify the instrument itself, because commercial tool specific frameworks (HP QTP, Ranorex, TestComplete) are usually a completely built and deployed infrastructure with actions emulator, scripting IDE and reporting.

Project specific frameworks

These frameworks are developed to implement automation during specific application development – to support specific application test automation requirements. Components of such frameworks can be constructed out of other open source libraries to form a specific environment around the SUT to support some or all of the following functions:

  • Deployment of the built application (along with its components such as database, services, backend) to an environment;
  • Starting the application;
  • Executing the test cases;
  • Reporting test run results directly into a test management system;
  • Control wrappers to support easier coding of automation using specific controls (grids, custom controls), etc.

Another important component to be considered is a harness to support running the test cases on different cloud environments to allow variability of supported OSs and browsers. Those can be home grown or parts of tool frameworks.

Best practices frameworks

Framework is so nice word that when you say it – it makes an impression. For instance, the Zachman Framework is not related to any developed components – it’s a methodology to define enterprise architectures. The same goes for in-house automation built frameworks – they can contain both components for test automation along with approaches, describing how to automate something in the best possible way. This is what test automation experts (including me) show to their customers willing to start test automation for the first time or to understand what is going on their current automation project.

Keyword driven frameworks

One more important framework family to mention. Those are tool specific or project specific frameworks and target less coding experienced staff – to enable them write and support automation scripts. The coded keywords (like Login, Click, NavigateToPage, TypeText) are implemented somewhere in a codebase behind the scenes as a keyword repository.

The keyword reference is provided to a tester so they can write scripts directly in a spreadsheet. Then these spreadsheets are fed to some keyword interpreter and a specific implementation from a repository is called to execute the test.

Test automation solution

I prefer to call test scripts along with all the underlying frameworks for a specific project a test automation solution. It includes all the specific/general frameworks for the project, test environment management, test solution architecture and best practices. And it includes test scripts. So when giving estimates – I prefer to speak about test automation solutions.

So where is the boundary?

Implementing test automation is always a significant investment. It is important to understand and clarify the return on investment as early as possible, otherwise the project can be just canceled.

Yes, there are occasions when you need to develop a specific test harness which takes time. But it’s just inexcusable to implement that separately from the test scripts. Automating test scripts with best practices in mind (for further maintainability) is of primary importance – it actually allows you to free some of the project’s time.

A real life example I’ve assessed was a “first try” automation project that was canceled because all efforts where spent to develop an environment management lab to support a variety of OSs where automation had to be executed afterwards. Experienced staff spent months developing it, but management didn’t see any ROI during this development – the manual testing cycles where the same.

So wouldn’t it be better to support only one or two of the highest priority environments with manual test deployment first and automate a test execution later on?

As long as you decrease the testing budget (or at least expect to) and deliver maintainable automated scripts – your budget owner is happy. The test harness behind the implementation might be huge and extra important for running those test cases, but let’s face it – you hardly need all the harness to start the first scripts. So I’d say it’s much more important to provide consistent test automation solution architecture (that would allow extending the harness in the future), than develop it completely upfront.

A good example of extendable test automation architecture is described here as a multi-layered test solution. Structuring the code into independent layers and forming Page Objects does not take much of the time but brings great maintainability for the final solution. And it really doesn’t matter if the final solution can be executed on one OS and web browser or several – multiplatform support can be added later on. 

Another important area to speak about is keyword driven frameworks. The approach dictates that you first need to develop a framework (a set of keywords), and afterwards you’ll be able to develop test scripts as chains of those keywords linked together.

I personally consider such approach to be a bad practice. First of all, development within a spreadsheet is too error prone. Any mistype could cause a problem which is hard to debug. Also, writing business logic without tests using the logic at once is pretty similar to developing an application’s business logic without unit tests or without even a user interface. And you never can estimate how many keywords you need to develop upfront before specific tests so they are sufficient for test scripts afterwards.

One approach

Let me describe my own approach to planning a test automation solutions. It’s proven to be the right one based on my own experience, but it’s one of many “the right ones”.

  1. What are the customer expectations from introducing test automation? Is it actually applicable at all within the project schedule and technology? I should say that sometimes the right answer at this stage is “test automation is completely not applicable to achieve your goal”. This can happen when automation development efforts would be too large in comparison to the achieved benefits.
  2. It is important to understand the technology of the system that is supposed to be tested and select the appropriate automation instrument tool for emulating user actions.
  3. What should the approach be? I distinguish two major approaches – “Quick and Dirty” and “Solution based”.

While “Solution Based” was described above, “Quick and Dirty” means it is valid to say “it works only on my machine”. With minimal investment, you can get results at once. It can be a good approach for Performance testing, when getting the performance characteristics of the system (sometimes, not always) is a one-time activity.

  1. Calculate the actual ROI of the implementation. Have a small proof of concept, time to run the manual regression/smoke test cycle and a number of those run cycles per release - we can understand the time (which is money) benefits.
  2. Propose a stage- based roadmap for test implementation. For each stage my main deliverable is automated test scripts, and the framework features are only added to facilitate the optimal and expected execution of those scripts.

Conclusion  

When doing estimates I prefer to speak about test automation solutions. Now, in an agile world, parts of the applications are delivered working and consistent, so why shouldn’t we do the same for test automation? Say no to developing only the frameworks! Do incremental deliverables (subsets of automation scripts along with those parts of the harnesses that are actually needed to run those scripts) to provide the best and fastest ROI.

About the Author

PhD, Test Automation Expert, TA Competence manager at SoftServe Inc.,Test Automation evangelist and consultant working both with small startups and fortune 500 companies on establishing the right quality assurance culture within both agile and waterfall product development environments. Speaker and trainer. Creator of  StepsToReproduce  - free tool for exploratory testing that can replace Windows Problem Steps Recorder. 

 

Rate this Article

Relevance
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Testing, especially automation is tricky by Ganesh Venkataraman

Sasha,

Thank you for the nice article. Frameworks in IT is heavily misused in many circumstances and Testing is not just left out. However, it is not completely grey. Let me take a specific scenario. In mobile app testing (read automation), if you create a regression script for say a particular view, if you are able to run the script on many operating systems - then we are talking about automation.

Think of a scenario where we need to test just the layout and the positioning of the elements in one particular screen size. If we could write out script that will test the view against a reference image by comparing the snapshot and ignoring all the actual data - then we are talking about a framework. There are so many approaches to test a mobile view depending on whether the app is native, mobile web or hybrid, considering the devices - phone and tablet, orientation and many other combination.

If we know exactly how we are going to test a piece given the parameters, then we are talking about framework. And automation can be a way to implement your framework. As I mentioned, it is definitely tricky and not all the bid teams get it. For them testing is, well, just testing. Thank you again for the article.

- Ganesh

Re: Testing, especially automation is tricky by Sasha Reminnyi

Hi Ganesh,

Thank you for your example!
Yeah, I agree - as long as there might be re-usability of artifacts - we can call this framework. That's why there are so many species)

- Sasha

Test automation is more than test execution by Maciej Gawinecki

> Automation tools/instruments are used to interact with the UI – emulate button clicks, type text, verify text box values.

I think this is just an example of what automation tools do in testing. What do you think about looking at automation wider? Automation can help also with interacting with services, queues, etc. It can be used not only to emulate a user or a client using the system but also for system configuration, generating test data (e.g., random email messages), simulating external systems, etc. There is a nice article "Agile Test Automation" of James Bach (www.satisfice.com/articles/agileauto-paper.pdf)explaining what other things automation tools can help in testing. See particularly "Test automation is more than test execution" section.

Test automation is more than test execution by Maciej Gawinecki

> Automation tools/instruments are used to interact with the UI – emulate button clicks, type text, verify text box values.

I think this is just an example of what automation tools do in testing. What do you think about looking at automation wider? Automation can help also with interacting with services, queues, etc. It can be used not only to emulate a user or a client using the system but also for system configuration, generating test data (e.g., random email messages), simulating external systems, etc. There is a nice article "Agile Test Automation" of James Bach (www.satisfice.com/articles/agileauto-paper.pdf)explaining what other things automation tools can help in testing. See particularly "Test automation is more than test execution" section.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

4 Discuss
General Feedback
Bugs
Advertising
Editorial
Marketing
InfoQ.com and all content copyright © 2006-2016 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT

We notice you’re using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.