BT

Use Test Categorization for Agile Builds

| by Deborah Hartmann Preuss Follow 0 Followers on Nov 02, 2006. Estimated reading time: 2 minutes |
Everyone agrees that developer testing is important, but as a test suite grows the time required to run those tests can actually become a team bottleneck.  This can discourage the full and frequent use of the test suite, resulting in increased risk of rework as the developer feedback loop lengthens.  IBM's DeveloperWorks has just published an article by Andrew Glover that reveals how categorization of tests can be used to help ensure end-to-end system soundness while keeping build times reasonable, even with today's massive test suites.

Many Agile teams have been encouraged to write xUnit tests from day one and run them as often as possible, using an automated "continuous" build process.  Teams often start with a nightly build and progress to running the entire suite at every developer check-in, using CruiseControl or similar tools, to provide fast feedback to developers.  This fast feedback is critical for Agile teams, many of whom work on very short timelines - delivering working code every week or month.  But as the suite grows and build times lengthen, this fast-feedback advantage predictably degrades, leaving teams with some tough choices: run a partial suite? go back to nightly builds? or take the to refactor the test suite to retain both build speed and test coverage?  Having a stategy can make the latter choice less painful.

Andrew Glover is a testing and security consultant whose company helps business address software quality by enabling teams to monitor code quality early and often.  He points out that any suite of unit tests is probably composed of unit tests AND component tests AND system tests.  Looked at this way, some teams will discover that the reason the build takes three hours is that the majority of them are component tests. His solution: categorization of tests and a rationalized build process.

Using jUnit and Ant examples, Glover explains his strategy, which includes creating custom JUnit suite files corresponding to desired categories, and custom directories for each type of test.  One example suggests: refactoring a team's Ant build file using categories, and modifying CruiseControl to only run the true unit tests on check-ins and component tests on an hourly basis, then creating an additional task that runs component tests and system tests together, hourly.

The end result of such a strategy is that tests are run many times a day and the team is able to catch integration errors more quickly -- normally within a few hours, which plays a vital role in ensuring code quality.

For examples and explanations, read the full article on IBM DeveloperWorks: In pursuit of code quality: Use test categorization for agile builds.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Categorization of Tests by Geoffrey Wiseman

Largely agree; although I might be inclined to organize my categories in a slightly different way; for instance, isolated unit tests (plentiful, cheap to run), persistence tests (require database, and infrastructure to use database, slower to run), acceptance tests (end-to-end, from UI through to db), etc.

That said, the point's still the same.

Test grouping - supported natively in TestNG by Alex Popescu

TestNG supports test grouping (and more related features like: group dependencies, groups of groups, fixture for groups, running individual groups without the recompile step, etc) for a long time.

I would also point out that "organization based" grouping has an immediate limitation: it is difficult to include a test in more than a group.

I would encourage everyone needing this feature to take a look at TestNG documentation and see how simple things are.

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

Re: Test grouping - supported natively in TestNG by Deborah Hartmann

Yes, I attended Andrew's session on TestNG recently, and was impressed by what TestNG adds. The many-to-many relationship of tests and suites is a good addition. Andrew only mentions it in a sidebar but his blog has a whole section on TestNG, including a comparison of JUnit and TestNG.

Frontline and backline tests by Slava Imeshev

Good article. This is exactly what we recommend Parabuild users - once the test set grows beyond 5 minutes, split it into a minimum set of tests that run as a part of the continuous integration cycle (frontline) and the set that run on a scheduled basis - hourly, daily (backline).

The frontline test set is essentially a smoke test that ensures that the system is alive after new changes are checked in. The backline test set gives the full picture of the system quality. This may also include full-blown automated unit tests.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

4 Discuss
BT