Use Test Categorization for Agile Builds

Everyone agrees that developer testing is important, but as a test suite grows the time required to run those tests can actually become a team bottleneck.  This can discourage the full and frequent use of the test suite, resulting in increased risk of rework as the developer feedback loop lengthens.  IBM's DeveloperWorks has just published an article by Andrew Glover that reveals how categorization of tests can be used to help ensure end-to-end system soundness while keeping build times reasonable, even with today's massive test suites.

Many Agile teams have been encouraged to write xUnit tests from day one and run them as often as possible, using an automated "continuous" build process.  Teams often start with a nightly build and progress to running the entire suite at every developer check-in, using CruiseControl or similar tools, to provide fast feedback to developers.  This fast feedback is critical for Agile teams, many of whom work on very short timelines - delivering working code every week or month.  But as the suite grows and build times lengthen, this fast-feedback advantage predictably degrades, leaving teams with some tough choices: run a partial suite? go back to nightly builds? or take the to refactor the test suite to retain both build speed and test coverage?  Having a stategy can make the latter choice less painful.

Andrew Glover is a testing and security consultant whose company helps business address software quality by enabling teams to monitor code quality early and often.  He points out that any suite of unit tests is probably composed of unit tests AND component tests AND system tests.  Looked at this way, some teams will discover that the reason the build takes three hours is that the majority of them are component tests. His solution: categorization of tests and a rationalized build process.

Using jUnit and Ant examples, Glover explains his strategy, which includes creating custom JUnit suite files corresponding to desired categories, and custom directories for each type of test.  One example suggests: refactoring a team's Ant build file using categories, and modifying CruiseControl to only run the true unit tests on check-ins and component tests on an hourly basis, then creating an additional task that runs component tests and system tests together, hourly.

The end result of such a strategy is that tests are run many times a day and the team is able to catch integration errors more quickly -- normally within a few hours, which plays a vital role in ensuring code quality.

For examples and explanations, read the full article on IBM DeveloperWorks: In pursuit of code quality: Use test categorization for agile builds.