Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Sauce Labs Adds Analytics and Extended Debugging to Continuous Testing Cloud

Sauce Labs Adds Analytics and Extended Debugging to Continuous Testing Cloud

At their recent user conference SauceCon, Sauce Labs introduced new features for its continuous testing cloud including test analytics, featuring a dashboard that analyses test results and exposes common failures by browser and operating system, including Android and iOS.

The builds table in the analytics dashboard includes a metric, "efficiency", that indicates the level of parallelisation achieved by the tests in each build. The efficiency metric is expressed as a percentage, as it is based on how long a build took to run as compared to the duration of the longest test within it.

Test trends provides visibility into tests, pass/fail/error trends, and test history and details of the results of a single test over time in order to identify unstable tests with inconsistent test run time and pass/fail rates. The GUI filters from an overview of all tests down to 5 second granularity.

Test insights test performance visualisation displays a scatter plot of each test run for that test using the time period and other filter criteria selected. Six performance statistics are provided for the test: total runs, total errors, total failures, fastest runtime, slowest runtime and average queue time. The scatter plot also shows each instance of the test being run, with colour coding to indicate the run status, against the time it took the test to either execute or fail. The X-axis indicates the time range that has been selected using the time filter. The Y-axis indicates the duration of the test each time it was run. Specific information about the platform, operating system, and other capabilities specified in a test is displayed.

A new Rest API (currently still in beta) allows users to aggregate the analytics data into their in-house built quality dashboard or test analytics tools and generate a unified view of all their test data. Existing Sauce plugins are available to integrate the testing platform into a continuous integration or delivery capability or DevOps toolchain.

Extended debugging offers a deeper understanding of test failures by surfacing network and browser errors helping developers to pinpoint issues. By adding the extendedDebugging capability to a Selenium test script, it becomes possible to collect both the JavaScript console logs (JS) and HTTP Archive Format (HAR) files generated by a test to use in debugging the test and build performance.

The JS console logs are a collection of security errors, warnings, and messages that are explicitly logged by the browser. These logs can be used to find out information like which components of a web application failed to load or ran into an error, what warnings were logged by the browser, and other application performance information. The console log information is associated with the URL where it occurred, and is composed of four types of information: Log, Info, Warning, Error.

HAR files are a collection of network requests and responses the browser makes and receives during the execution of a test. These files provide information like which browser requests have timed out or never loaded, which requests are slowing down the loading process, and API calls that are having errors. Since tests will generate additional assets when the extendedDebugging feature is used there may be an impact on test performance. This feature is currently only available for testing on Chrome and Firefox browsers and now for Android emulators, is available to all Sauce Labs users at no additional cost and generally available now.

The full test analytics feature set will be generally available in the second quarter of 2018 and will be provided to Sauce Labs Enterprise customers at no additional cost.

Rate this Article