Targeted Auto Retry is Ebay's approach to dealing with test flakiness that aims to make a continuous integration pipeline more resilient to flaky test steps. To make this approach straightforward to use, Ebay has open sourced a lightweight framework for the Swift language that can be used with Xcode unit testing framework.
Flaky tests are tests that could fail due to any kind of reason not directly related to a bug in your app code. Typical causes of flakiness are, for example, timing issues on app launch, data caching, incorrect test setup, failing network requests, some subtle bug in the test implementation, and so on.
A flaky test could block/delay development until spotted and resolved. The problem is that you do not know if you caused the test failure or if it is flaky. There is no easy way to deal with flaky tests.
Two common ways to deal with flaky tests are running all failing tests again and placing suspect tests in a quarantine. By running the tests again, you can identify which tests fail each time, thus pointing to a bug in your code, and which ones are flaky and require further analysis. All tests suspected to be flaky are placed in a quarantined areas so they will not stop your CI pipeline until investigated.
As Ebay engineer Evan Pierce explains, re-running all failing tests has a number of disadvantages, including the possibility of missing on intermittent bugs, the need for some specific tool to automate this step, slowness, and more. Instead of running whole tests, Targeted Auto Retry focuses on those steps that are more likely to fail, such as app launch.
In practice, with Targeted Auto Retry, you focus on some step of your setUp/tearDown unit test and, if that step fails, you run it again until either it passes or a maximum retry limit is reached. This is shown in the image below:
It is important to notice that Targeted Auto Retry is not applied to all of your unit tests by some tool able to instrument your test code. Rather, it is your choice to add it for a specific step of a test you suspect could be flaky. In this way, it provides a convenient way to further investigate a failing step.
This approach, says Pierce, has a number of advantages over running failing tests from scratch, including less risk of missing on intermittent bugs, faster execution, and simple implementation.
Mobile teams at eBay that have adopted it have seen their flaky test problems virtually disappear overnight – with up to 99% of flakiness eliminated from their entire regression test suites.
The open source TargetedAutoRetry package is specifically aimed to Xcode and the Swift language. Once you have added it to your project as a dependency, if you want to use Targeted Auto Retry for a XCTestCase
of yours, you only need to make it conform to the TargetedAutoRetry
protocol by implementing the autoRetry
method, e.g.:
autoRetry(
mainAction: { XCUIApplication().launch() },
successCondition: { XCUIApplication().buttons["Foo"].exists },
actionDescription: "Launch"
)
Here you can check out more examples of using Targeted Auto Retry with Xcode unit tests.