Frequently during the early stages of test automation project consulting I am asked for estimates around automation implementation. And when I first give the numbers the next question – “is that for a test suite or for a framework?”
This question always makes me a bit stuck – I don’t know what I’m being asked for… What are test scripts? What is the framework? Can they be separated?
Let’s start with some definitions.
Automation tools
Automation tools/instruments are used to interact with the UI – emulate button clicks, type text, verify text box values. At least here I can say for sure – there is no ambiguity.
Frameworks
Some time ago my vision about a framework was rather specific –reusable, SUT (system under test) agnostic and automation tool agnostic libraries to speed up automation implementation.
But in the IT world the word framework is so frequently used, that even when speaking about automated testing only it has different meaning depending on the context.
Pic 1 –different frameworks examples
Tool specific frameworks
Commercial automation tool vendors or open source communities frequently develop a whole infrastructure around their tools that allow generating reports, running test suites, distributed test execution, working with test execution environments. For instance, the Selenium tool’s main component is WebDriver – which works like a plugin for a web browser to manipulate the DOM model (xml-based document object model of the web page) of the web application from inside the web browser. But there are also a number of additional coding libraries, and a record-playback tool (Selenuium IDE). All of those together form the Selenium test automation framework.
Another great example would be Serenity that is also tool specific (built around Selenium Web Driver) framework. But its goal is not to have many optional components/plugins. It can be considered as an accelerator – specific components are already combined by the community experts into one substance - to support the possibility of faster test automation implementation startup and also support for BDD approaches.
In the commercial world, it’s even harder to identify the instrument itself, because commercial tool specific frameworks (HP QTP, Ranorex, TestComplete) are usually a completely built and deployed infrastructure with actions emulator, scripting IDE and reporting.
Project specific frameworks
These frameworks are developed to implement automation during specific application development – to support specific application test automation requirements. Components of such frameworks can be constructed out of other open source libraries to form a specific environment around the SUT to support some or all of the following functions:
- Deployment of the built application (along with its components such as database, services, backend) to an environment;
- Starting the application;
- Executing the test cases;
- Reporting test run results directly into a test management system;
- Control wrappers to support easier coding of automation using specific controls (grids, custom controls), etc.
Another important component to be considered is a harness to support running the test cases on different cloud environments to allow variability of supported OSs and browsers. Those can be home grown or parts of tool frameworks.
Best practices frameworks
Framework is so nice word that when you say it – it makes an impression. For instance, the Zachman Framework is not related to any developed components – it’s a methodology to define enterprise architectures. The same goes for in-house automation built frameworks – they can contain both components for test automation along with approaches, describing how to automate something in the best possible way. This is what test automation experts (including me) show to their customers willing to start test automation for the first time or to understand what is going on their current automation project.
Keyword driven frameworks
One more important framework family to mention. Those are tool specific or project specific frameworks and target less coding experienced staff – to enable them write and support automation scripts. The coded keywords (like Login, Click, NavigateToPage, TypeText) are implemented somewhere in a codebase behind the scenes as a keyword repository.
The keyword reference is provided to a tester so they can write scripts directly in a spreadsheet. Then these spreadsheets are fed to some keyword interpreter and a specific implementation from a repository is called to execute the test.
Test automation solution
I prefer to call test scripts along with all the underlying frameworks for a specific project a test automation solution. It includes all the specific/general frameworks for the project, test environment management, test solution architecture and best practices. And it includes test scripts. So when giving estimates – I prefer to speak about test automation solutions.
So where is the boundary?
Implementing test automation is always a significant investment. It is important to understand and clarify the return on investment as early as possible, otherwise the project can be just canceled.
Yes, there are occasions when you need to develop a specific test harness which takes time. But it’s just inexcusable to implement that separately from the test scripts. Automating test scripts with best practices in mind (for further maintainability) is of primary importance – it actually allows you to free some of the project’s time.
A real life example I’ve assessed was a “first try” automation project that was canceled because all efforts where spent to develop an environment management lab to support a variety of OSs where automation had to be executed afterwards. Experienced staff spent months developing it, but management didn’t see any ROI during this development – the manual testing cycles where the same.
So wouldn’t it be better to support only one or two of the highest priority environments with manual test deployment first and automate a test execution later on?
As long as you decrease the testing budget (or at least expect to) and deliver maintainable automated scripts – your budget owner is happy. The test harness behind the implementation might be huge and extra important for running those test cases, but let’s face it – you hardly need all the harness to start the first scripts. So I’d say it’s much more important to provide consistent test automation solution architecture (that would allow extending the harness in the future), than develop it completely upfront.
A good example of extendable test automation architecture is described here as a multi-layered test solution. Structuring the code into independent layers and forming Page Objects does not take much of the time but brings great maintainability for the final solution. And it really doesn’t matter if the final solution can be executed on one OS and web browser or several – multiplatform support can be added later on.
Another important area to speak about is keyword driven frameworks. The approach dictates that you first need to develop a framework (a set of keywords), and afterwards you’ll be able to develop test scripts as chains of those keywords linked together.
I personally consider such approach to be a bad practice. First of all, development within a spreadsheet is too error prone. Any mistype could cause a problem which is hard to debug. Also, writing business logic without tests using the logic at once is pretty similar to developing an application’s business logic without unit tests or without even a user interface. And you never can estimate how many keywords you need to develop upfront before specific tests so they are sufficient for test scripts afterwards.
One approach
Let me describe my own approach to planning a test automation solutions. It’s proven to be the right one based on my own experience, but it’s one of many “the right ones”.
- What are the customer expectations from introducing test automation? Is it actually applicable at all within the project schedule and technology? I should say that sometimes the right answer at this stage is “test automation is completely not applicable to achieve your goal”. This can happen when automation development efforts would be too large in comparison to the achieved benefits.
- It is important to understand the technology of the system that is supposed to be tested and select the appropriate automation instrument tool for emulating user actions.
- What should the approach be? I distinguish two major approaches – “Quick and Dirty” and “Solution based”.
While “Solution Based” was described above, “Quick and Dirty” means it is valid to say “it works only on my machine”. With minimal investment, you can get results at once. It can be a good approach for Performance testing, when getting the performance characteristics of the system (sometimes, not always) is a one-time activity.
- Calculate the actual ROI of the implementation. Have a small proof of concept, time to run the manual regression/smoke test cycle and a number of those run cycles per release - we can understand the time (which is money) benefits.
- Propose a stage- based roadmap for test implementation. For each stage my main deliverable is automated test scripts, and the framework features are only added to facilitate the optimal and expected execution of those scripts.
Conclusion
When doing estimates I prefer to speak about test automation solutions. Now, in an agile world, parts of the applications are delivered working and consistent, so why shouldn’t we do the same for test automation? Say no to developing only the frameworks! Do incremental deliverables (subsets of automation scripts along with those parts of the harnesses that are actually needed to run those scripts) to provide the best and fastest ROI.
About the Author
PhD, Test Automation Expert, TA Competence manager at SoftServe Inc.,Test Automation evangelist and consultant working both with small startups and fortune 500 companies on establishing the right quality assurance culture within both agile and waterfall product development environments. Speaker and trainer. Creator of StepsToReproduce - free tool for exploratory testing that can replace Windows Problem Steps Recorder.