Faster Test Runs With Clover's Test Optimization

The recent release of Clover 2.4 highlights a new "Test Optimization" feature that offers to speed up CI builds and allow developers to spend less time waiting for their tests to run. The feature leverages "per-test" coverage data to selectively run only the tests impacted by your code changes.

Atlassian has just released the 2.4 version of their popular code coverage analysis tool Clover, adding a new feature dubbed "Test Optimization". From Atlassian:

Clover has the ability to optimize test runs, greatly reducing the time it takes to test a code change. Typically, the full suite of tests has run whenever a code change is made. With Test Optimization enabled, Clover automatically determines the optimal subset of tests to run based on the specific changes made. Testing only what you need provides quicker feedback without compromising test quality.

Reducing the time it takes to understand the impacts your code changes have made on your regression test suite could offer many teams significant improvements in productivity. It is important to note that many people will argue that is why team's must strive to keep their unit tests lightning quick, and this is absolutely true. For many reasons though, even if the team has made each unit test lightning quick, the aggregate of their entire application's test suite may still take longer to run than is optimal.

A logical approach to improving this is to selectively run only the tests affected by a given code change. Doing this manually not only takes a decent amount of work, but it often leads teams to "miss tests" rather frequently, ultimately losing the benefit of the optimized test run. This new clover feature offers a way for teams to take this approach without the manual effort and with a lowered risk of missing a test that should have been run but wasn't.

Brendan Humphreys describes more about how Clover does this:

As a code coverage tool, Clover measures per-test code coverage - that is, it measures which tests hit what code. Armed with this information, Clover can determine exactly which tests are applicable to a given source file. Clover uses this information combined with information about which source files have been modified to build a subset of tests applicable to a set of changed source files. This set is then passed to the test runner, along with any tests that failed in the previous build, and any tests that were added since the last build.

According to Humphrey's, Test Optimization also makes it possible to be more strategic about the order that the tests are run, which he claims can improve test run effectiveness. About these strategies:

The set of tests composed by Clover can also be ordered using a number of strategies:

• Failfast - Clover runs the tests in order of likeliness to fail, so any failure will happen as fast as possible.
• Random - Running tests in random order is a good way to flush out inter-test dependencies.
• Normal - no reordering is performed. Tests are run in the order they were given to the test runner.

Humphreys goes on to describe the results of 10-day trial by their FishEye team, where he states that their "test execution time was reduced by a factor of four".

Take a moment to read up on this new Clover release, particularly Humphrey's take, to see if this can help your team.