Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Continuous Quality and the Cloud: How You Should Be Testing Mobile Apps

Continuous Quality and the Cloud: How You Should Be Testing Mobile Apps

”Mobile First,” a new strategy for customer engagement, is becoming widely adopted throughout many organizations. At the same time, mobile is driving agile as the mobile app market changes its complexity daily and mobile app Dev/Test teams need to meet higher velocity of releases. The new reality is that mobile apps need a different development process.

Mobile app developers are now required to cope with a growing set of changes in the lifecycle to keep pace with the market—from end users with new mobile platforms and devices to the growth of the digital experience for end user engagement. For instance, Internet of Things (IoT) involves more context-aware devices with new sensor-based capabilities and the need to integrate with desktops, phones, wearables, tablets, and non-mobile devices. With these changes, we need to accept that mobile will continue to be disruptive; and constantly introduce new challenges around functionality and performance of mobile apps.

As mobile apps push innovation and requirements and functionality evolve, the complexities associated with testing these apps grow in parallel. How do you ensure that the garage-door opening app not only works, but works well and works consistently? And how do you achieve greater velocity in developing this app, without compromising its quality? The only answer to support mobile app developer’s quest for improved velocity and quality is to embed testing from the early stages of the mobile SDLC throughout the lifecycle release to production. This is the Continuous Quality methodology.

Continuous Quality

Continuous Quality is a methodology for embedding quality activities into every step of the SDLC process— from design through build to production— all based on supporting processes, tools and testing lab infrastructure that is customized to support an organization’s specific requirements. A successful Continuous Quality process optimizes time to market, drives faster and more frequent releases and enables minimizing escaped defects to production by managing risk in an automated way as early as possible.

The main driver behind the rise of Continuous Quality is the constant need to release high quality mobile apps (Native, Hybrid, etc.) more frequently. For mobile, having agile-ready processes is necessary, but on top of the agile methods, development teams need fast and actionable feedback from real devices on both the functional aspects of their app as well as the performance, usability and security. Embedding all of the quality aspects into each build and delivering constant feedback to the development team gives them early insights into bugs - this is the core value of continuous quality.

Today, thousands of tests need to run against an average enterprise mobile app. These processes ought to be tested on variety of both Android and iOS devices, smartphones and tablets and additionally at least two or three of the latest operating systems versions.

An optimal Continuous Quality process relies on a Continuous Integration (CI) server that uses test automation to run scripts across a device test matrix, made up of these Androids, iOS devices, tablets, phones and different OS versions. The development team obtains reports from the CI server, where the length of such a cycle would be measured in hours (overnight). In a broken automation process, it’s near impossible to transform a quality process into a repeatable cycle, and instead often turns into a lengthy and inefficient one.

Failing to Succeed: Working without Continuous Quality

Organizations that use old and traditional development methods can simply not succeed with mobile. A team who does not practice a Continuous Quality workflow will end up working in a disconnected manner using very different tools from the ones their developers use. When a team is still relying on a few mobile devices for testing on a developer’s desk or QA desks, and they do not host these devices in a cloud environment, they face more challenges in the app development cycle including the lack of governance and security. The team will heavily rely on manual and exploratory testing with some basic automation – a far cry from a continuous integration machine which can run thousands of tests overnight on multiple devices.

This team also lacks the ability to create and test true end-user conditions. There are tools that will mimic and allow developers to test on various network profiles, carrier networks, GPS location testing, load testing etc. The outcomes of such “old” environments are a limited visibility into the application quality and performance that often results in late and expensive defects identified by the field or customers.

Implementing Continuous Quality

The key differences between a Continuous Quality process and other processes are that during a Continuous Quality process, the barriers between developers, testers, and performance engineers are eliminated. The focus is put on quality from the very early stages through a set of environmental assets, tools and a quality lab that provides the developers with a natural environment where they develop, build and test their mobile app in parallel, as part of their continuous integration (CI) workflow.

In order to implement a Continuous Quality process for mobile apps, which is by far more fragmented, complex and hard to predict than web, mobile app developers or mobile agile teams need to have a few unique tool capabilities in order to have the strongest Continuous Quality process.

A dev team should consider a mobile app quality lab with real devices available to test on; emulators fall short in many technical aspects and cannot not provide full device capabilities such as sensors, network conditions, and specific hardware constrains, and more. The optimal setup is devices connected to real mobile networks in a cloud-based lab, a lab that can provide relevant capabilities for controlling mobile devices:

  • Full mobile device system control
  • Device rotation
  • Device logs and vitals such as CPU monitoring
  • Real network conditions where the developers can test their apps against various network profiles such as 2g, 3g, with different packet loss rates and many more, and inspect the app in various load conditions
  • Location based testing capabilities (e.g. – leverage Google maps API’s to mock a device location) as part of the early testing and through the CI workflow
  • Simple USB device connectivity, non-intrusive, non jail-broken devices
  • Simulating network environments
  • Environment switching capabilities to enable testing one app against staging and production environments easily

Leveraging the Hybrid Cloud

With so many changing requirements and a constantly-evolving mobile device market, teams can also benefit from a “hybrid cloud” environment. A hybrid cloud is a cloud environment made up of a variety of deployment options - organizations can work on a combination of device-in-hand, shared devices and remotely hosted mobile devices for testing, which complements relevant coverage requirements and enables collaboration between global teams.

With a hybrid cloud, elasticity helps teams grow and scale as mobile app projects expand or transform. It’s also valuable in meeting both local development use cases such as testing wearables which requires local pairing via Wi-Fi or Bluetooth, or testing NFC based apps like payment apps. Hybrid cloud also meets global device coverage requirements, where a distributed team requires collaboration with a remote team who holds unique devices in their specific network. Leveraging a hybrid cloud environment, said devices become part of the joint cloud where all teams share and develop or test on, regardless of location.

End-to-End SDLC support

It’s important that developers and testers speak the same language in order to meet the challenging release cycle targets. Test automation that works is a critical pillar in the Continuous Quality workflow. Such automation will enable both the development and testing teams to write efficient test code in whatever framework, language they work with, and seamlessly run it on multiple real devices as part of their overall CI and build acceptance processes. Such automation solution integrates with any CI process, test framework (Selenium, Appium, Calabash, etc.), and IDE (Eclipse, Visual Studio, Xcode, others) making it a foundation pillar for the Continuous Quality process.

Beyond just manual and functional testing, test environment should offer the non-functional performance and monitoring capabilities as part of the early development phases and CI. It is important to realize that apps often run in parallel with dozens of other apps in various network conditions, competing on memory constrains, CPU, battery stretch and network bandwidth. Therefore, it is really naïve to test your app in a “clean” room environment without real load, complex scenarios, incoming events and others. Not paying attention to the above as part of your overall Continuous Quality process exposes your app to late-in-the-game defects which are harder to analyze, fix and re-deploy especially when having to pass through the app store’s lengthy re-deployment processes.

The Developer Role In The Continuous Quality Process

The Continuous Quality process empowers developers and shifts the focus to building quality apps. Developers who are used to very basic unit tests and to handing their binary app (IPA, APK etc.) to the QA team now have the power and tools to make the entire Continuous Quality cycle more efficient and with higher test coverage.

Developers can inject a large set of validations into their on-demand app build (daily, hourly, etc.) and get a comprehensive report in a matter of hours; it’s a game changer for them. As part of the build, developers can include robust and durable test automation written either by them or by the automation engineers in the same environment and language they know. Teams should define the most important test cases with clear criteria (most uncover defect test cases, tests for key features, customer specific bug fixes and more) and include them in every CI regression cycle. Once these tests pass, the remaining test cases launch for higher coverage and risk reduction.

On top of the functional automation done against real devices in real network conditions, the developers can include very important and relevant “environment testing” capabilities to increase their validation coverage and get the maximum insights in order to enhance their confidence in the app they are about to release to the market.

The Changing Market: Wearables Next

As wearables are now changing and speeding up the already fragmented market even further the need for a Continuous Quality process as part of the current SDLC is stronger than ever.

Wearables apps and mobile apps will constantly be interacting with each other, and will depend on each other. Actions from users happening in real time on one device, will trigger other actions and events on the second device. When a person finishes his/her run while using a FitBit, they’ll immediately look to their mobile device for the latest stats and comparisons to the prior day. To cope with this, developers will need to be able to have full control over the device (smartphone/tablet) system as well as on the wearable device. With a robust Continuous Quality process in place, a developer has the full control on their mobile environment and on the tools used to develop and test apps, as well as the support for such additional platforms.

Making the leap towards a Continuous Quality Process: One Financial Institution’s Journey

How do we make the change and move from a leggy and inefficient mobile app development process into a top-notch Continuous Quality one?

A top financial institution was suffering from a lengthy QA cycle of 8 weeks for a single release. The cycle was not only expensive, but also limited in scale, and as a result, related risks grew with every build. Poor device coverage and limited insights per build caused developers to act slowly and respond to the market end users.

In that time, the financial institution did not have feedback integrated into the process, which made defects reported by the field take days just to get reproduced and investigated. End users today don’t have the time, patience or desire to wait for such fix, they move to the competitor.

The financial institution realized reacting to mobile app quality based on public customer feedback through app stores was not an option and they need to be connected to the market to keep customers. This institution set clear goals for having more frequent releases, which meant having higher percentage of test automation that included functional tests and performance testing.

To transition to a Continuous Quality development process and improve the speed of which the app was released, this organization made critical changes in its mind-set. They asked some tough questions – where did they want their app to be in six months, a year year and more from today? How should their release cycle look like from quality perspective? How were they going to increase efficiency? Most importantly, how would their end user experience continue to suffer without making these changes?

After some research and analysis, the following goals were determined:

  • Improve app quality with app store rating growing to 4-5 stars
  • Reduce the release cycle length with granular scope – run automation tests according to impacted code areas
  • Increase efficiency with an automation goal was set to 80 percent, thus increasing time to market and fast MTTR (mean time to resolution) of defects.
  • Grow Device coverage and include their target market top leading devices

The Results

To assess a mobile application today and make sure it works on desktop browsers, mobile devices and more, the financial institution realized that its developers needed to incorporate testing throughout the entire mobile app development lifecycle. This would improve the quality of the app and improve development velocity. They needed a testing lab that is always available to them from the beginning of mobile app development. One that would grant access to the devices needs at any time, in a fully charged state, with constant connection to real carrier networks and with full control over the device system and logs. By testing in this kind of environment, they were able to detect issues as early as possible in the development lifecycle – obviously key to improving both development velocity and the quality of the mobile app.

Additionally, with a hybrid cloud test environment, teams no longer needed to chase devices or change languages for testing. They were free to focus on the release quality and device coverage.

By implementing a Continuous Quality process complete with test automation and a cloud-based lab, the financial institution was able to automate 80% of their manual tests and integrate the testing into the CI workflow to be executed on up to 20 real devices, on top of this, they were able to add real devices performance testing on both Android and iOS apps in real network conditions – all leveraging the Hybrid Cloud model mentioned above.

To summarize the above here is a “cook book” to make the change and move to a continuous quality process:

The common use of automated release management tools like Jenkins, combined with automated functional regression testing for mobile apps is clear and a key for mobile apps velocity & quality, however and especially in the mobile environment - performance testing must also be part of this automated release procedure. Code changes that may degrade performance may be introduced without notice. Implementing automated performance measurements as part of the automated build eliminates this risk and gives early insights to the developers and performance engineers.


With a new year of mobile upon us and the promise of new platforms, mobile app developers need to embrace new processes and methodologies that can put them in an enhanced position in the organization where they can respond faster to the market changes.

A Continuous Quality process is the most efficient solution for organizations to address the market changes fast and with high and predictable results.

Developers should constantly “listen” to what the market has to say and how it will impact their development— either from new platforms releases, new devices coming into the market and can impact their device matrix, app store reviews and major market events which can impact the mobile app quality.

The bridge that Continuous Quality provides to developers and testers, makes the daily jobs in the mobile app SDLC more efficient since the entire team gets the insights from every CI build instantly, from real devices in real network conditions, on large scale of devices and OS’s and all goes directly to the developers natural environment, and this is key in today’s mobile space.

About the Author

Eran Kinsbruner is director of product marketing at cloud-based mobile application testing and automation company Perfecto Mobile. Formerly CTO for mobile testing at Texas Instruments, and project manager at Matrix, Eran has been in testing since 1999, with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and Neustar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has extensive experience in the mobile testing world. You can find him on Twitter, LinkedIn, and his professional mobile testing blog at Eran also writes regularly for the Perfecto Mobile blog.

Rate this Article