Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Scalable Automation Frameworks for Functional and Non-Functional Testing

Scalable Automation Frameworks for Functional and Non-Functional Testing

Separating the capabilities of a testing framework from the actual tests can enable scaling automated testing for complex enterprise products. According to Alexander Velinov, we should agree on the types of tests to execute automatically during release and what should be kept as manually triggered tests.

Velinov spoke about scaling continuous testing and automation at QA Challenge Accepted 2022.

Complex enterprise products consist of different products, and each one consists of many little solutions (services, DBs, SPAs). All these modules are living independently, and have their independent development and delivery process, Velinov said.

Imagine a picture of hundreds of development repositories that should be tested somehow, not only functionally, but also non-functionally. This requires something different from the usual monolith testing repositories with all tests included, Velinov explained:

We need a more flexible architecture where we separate the capabilities of the framework (example: Selenium wrapper, API library, etc) in one repository that is compiling a reusable "capability package," and keep the actual tests and all their related code into a separate repository consuming that "capability package".

We have a lot of options for tools and frameworks that could be used to automate the basic performance and security tests that we should include in our delivery pipelines, Velinov said. He mentioned that the key here is to plan it carefully, and as a team to agree on the types of tests that you would like to execute automatically during the release process, and which should be kept as manually triggered tests in a controlled environment.

Velinov suggested that, for accurate results, most of the automated tests should be done in an environment identical to the production one in terms of configuration and data (anonymized of course).

InfoQ interviewed Alexander Velinov about scaling automated testing.

InfoQ: What does a scalable functional automation architecture that supports micro-services and micro-frontends development and delivery look like?

Alexander Velinov: Imagine all the frameworks and library wrappers that you are using for your user interface testing (Selenium), API testing (RestSharp/REST Assured, etc), and DB testing into a single solution and package. Next, there will be many different "service testing frameworks" that cover your microservices.

The approach with capability packages can be followed for every shared step, particularly for common functionality or flow (like user management, onboarding process, etc). This way we prevent duplicating the same code in a lot of "service testing frameworks".

InfoQ: How can we do performance and security testing as part of a continuous delivery pipeline?

Velinov: I suggest considering automated baseline performance tests (quick enough, like a single load of 10 simultaneous users) that are executed on every code change and are comparing the performance metrics against the previous run. If we see a degradation, that should be investigated.

For security, you can quickly add passive scans, which are provided by many commercial tools, and also OWASP ZAP (open source), which are relatively quick and help you avoid security configuration issues. Also, most of the commercial security testing tools are offering static code analysis of your code for security vulnerabilities. This again is something you can configure automatically in your continuous integration process.

The rest of the performance testing types like load, stress, soak, etc, should be planned and executed in a controlled manner. By controlled, I mean their execution is manually triggered, into a dedicated environment (or shared but currently not used by anyone), since these tests are impacting and sometimes even blocking the environment performance.

The same approach can be used with the other important security testing type – the active scan, which pretty much tries to automatically penetrate the system and generates a lot of junk data into your DB. This would be your solid starting point for building mature delivery pipelines.

About the Author

Rate this Article