Test automation in mobile development should be done by the Scrum team; don’t set up separate test automation teams, said Nadya Denisenko at the European Testing Conference. She advised obeying the testing pyramid for mobile testing and involve testers from the start. Testers are quality-oriented developers who can guide and assist other developers in delivering high-quality software; manual testing will disappear in the future, she said.
Mobile development is vendor locked. There are two major vendors on a market that dictate how operating system, apps, development and testing should go, said Denisenko. In addition, most of the companies are looking for a test automation unicorns that can develop automated tests on both platforms. It means that automation engineers in mobile should know at least Kotlin, Swift, Java and Objective-C and how iOS and Android works, and they expect from automation engineer to be able to manually test the app, which is not the case, said Denisenko.
Mobile apps are known for the fast development and in most of the cases QA is involved in the very late stages of it when the product is on a market for quite some time, said Denisenko. This brings a ton of challenges with setting up testing processes in a team that got used to work for a long time without a tester, she stated. Her advice would be to start slow; together with developers build a test automation framework first, automate features taken into sprint and take one regression scenario.
Mobile projects are quite small comparing to web or backend; it doesn’t make sense to have a separate automation team for the tasks that can and should be taken care of by the Scrum team, said Denisenko.
Denisenko mentioned that the role of the tester is to guide and assist developers in delivering high-quality software. She stated, "I strongly believe that a tester is a quality-oriented developer and manual testing will disappear or transform in the future."
More and more companies want to have developers take care of testing by developing testable code and tests, said Denisenko. She speaks from her path of being a manual tester to becoming a test automation engineer, believing that the role of testers is transforming to software developers in test or quality coaches.
InfoQ is covering the European Testing Conference 2019 and spoke with Nadya Denisenko after her talk about ways to fail at mobile test automation and how to avoid it.
InfoQ: You stated in your talk that separate automation teams are a waste of money. Can you elaborate why?
Nadya Denisenko: A major reason is the design of testing. Most of us use the test pyramid with 70% unit tests, 20% integration tests, and 10% E2E automated tests when making decisions on test coverage. It’s very common in the mobile world to disobey the testing pyramid and end up with either testing hourglass or testing ice cream cone. In most of the cases, having a separate automation team means that the main focus of such a team is automating E2E tests so it makes more sense to distribute resources according to test design.
InfoQ: Does the testing pyramid apply or would an hourglass or ice cream be a better fit?
Denisenko: It does. The testing pyramid gives better control over what’s happening in the app and saves time on debugging an issue.
InfoQ: Why do mobile testers disobey the testing pyramid?
Denisenko: There are several reasons and all depends on the situation:
Silver bullet. Management and some developers (especially backend ones) think that by having E2E UI tests, running against the real environment in all real-life situations can be covered. In addition, they think that those tests will cover the absence of API tests, backend, and client integration tests, which is wrong. There are so many things that cannot be tested on mobile because of platform limitations. A simple example would be a deep linking from/to external apps and push notifications. People also tend to forget that there are too many layers between the backend and application’s UI where it all can go wrong and there are no frameworks I know of that can give detailed info of where exactly the problem was: 3rd party, backend, network, network implementation in the app, UI, you name it. As a result, projects ends up with unmaintainable tests and disappointment in test automation.
Timing. New mobile projects always start as an MVP and then grow to something big. It always starts without thinking about the app’s testability, meaning that the app is not designed for more than unit and E2E UI tests. By the time developers see a need for in-depth testing, it’s too expensive to make changes so teams just ignore.
Expertise. Sometimes it’s just a matter of expertise. Integration testing is a new wave in mobile testing and not every single developer has enough knowledge and understanding of what it is and how it has to be done. Some don’t even have a desire to learn it.
InfoQ: What have you learned when it comes to automating mobile testing?
Denisenko: I’ve learned several things:
- Never try to play a catch-up game when joining a project without automation.
- When developing a test automation framework, try to use the vendor’s testing frameworks if possible. Open source solutions tend to release support for the latest OS versions six months after the release (meaning that none of the automated tests will work on the new OS) and also they tend to be discontinued.
- Never try to adapt tests that were developed for another project. I ended up with continuous test integration and fixing tests the whole sprint instead of focusing on developing and maintaining my own tests.
- Use native languages. Developers are lazy; they will never learn ruby or python just for testing purposes.
- Quality is a shared responsibility and every team member should contribute to it.
InfoQ: What testing guidelines do Apple and Google provide, and how can we use them?
Denisenko: The testing guidelines are:
Google suggests to do testing on different levels: unit, integration (integration between components), UI testing, functional UI testing, E2E testing. Google tries to grow a generation of developers who know how to test their code on different levels preferably with test automation. They have created a lot of tutorials about that and Googles’ testing community is quite vocal and active.
Apple, however, encourages devs to develop unit tests and E2E tests. They suggest developers to automate an app as real users use it and cover E2E tests with automation.
In my opinion, vendors shouldn’t influence developers and testers, and let them decide which strategy is better. By adding rules and setting limits, they actually reduce the possibility that something new, better and innovative will be created.