Everyone agrees that developer testing is important, but as a test suite grows the time required to run those tests can actually become a team bottleneck. This can discourage the full and frequent use of the test suite, resulting in increased risk of rework as the developer feedback loop lengthens. IBM's DeveloperWorks has just published an article by Andrew Glover that reveals how categorization of tests can be used to help ensure end-to-end system soundness while keeping build times reasonable, even with today's massive test suites.
Many Agile teams have been encouraged to write xUnit tests from day one and run them as often as possible, using an automated "continuous" build process. Teams often start with a nightly build and progress to running the entire suite at every developer check-in, using CruiseControl or similar tools, to provide fast feedback to developers. This fast feedback is critical for Agile teams, many of whom work on very short timelines - delivering working code every week or month. But as the suite grows and build times lengthen, this fast-feedback advantage predictably degrades, leaving teams with some tough choices: run a partial suite? go back to nightly builds? or take the to refactor the test suite to retain both build speed and test coverage? Having a stategy can make the latter choice less painful.
The end result of such a strategy is that tests are run many times a day and the team is able to catch integration errors more quickly -- normally within a few hours, which plays a vital role in ensuring code quality.
Many Agile teams have been encouraged to write xUnit tests from day one and run them as often as possible, using an automated "continuous" build process. Teams often start with a nightly build and progress to running the entire suite at every developer check-in, using CruiseControl or similar tools, to provide fast feedback to developers. This fast feedback is critical for Agile teams, many of whom work on very short timelines - delivering working code every week or month. But as the suite grows and build times lengthen, this fast-feedback advantage predictably degrades, leaving teams with some tough choices: run a partial suite? go back to nightly builds? or take the to refactor the test suite to retain both build speed and test coverage? Having a stategy can make the latter choice less painful.
Andrew Glover is a testing and security consultant whose company helps business address software quality by enabling teams to monitor code quality early and often. He points out that any suite of unit tests is probably composed of unit tests AND component tests AND system tests. Looked at this way, some teams will discover that the reason the build takes three hours is that the majority of them are component tests. His solution: categorization of tests and a rationalized build process.
The end result of such a strategy is that tests are run many times a day and the team is able to catch integration errors more quickly -- normally within a few hours, which plays a vital role in ensuring code quality.
For examples and explanations, read the full article on IBM DeveloperWorks: In pursuit of code quality: Use test categorization for agile builds.
Community comments
Categorization of Tests
by Geoffrey Wiseman,
Test grouping - supported natively in TestNG
by Alex Popescu,
Re: Test grouping - supported natively in TestNG
by Deborah (Hartmann) Preuss,
Frontline and backline tests
by Slava Imeshev,
Categorization of Tests
by Geoffrey Wiseman,
Your message is awaiting moderation. Thank you for participating in the discussion.
Largely agree; although I might be inclined to organize my categories in a slightly different way; for instance, isolated unit tests (plentiful, cheap to run), persistence tests (require database, and infrastructure to use database, slower to run), acceptance tests (end-to-end, from UI through to db), etc.
That said, the point's still the same.
Test grouping - supported natively in TestNG
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
TestNG supports test grouping (and more related features like: group dependencies, groups of groups, fixture for groups, running individual groups without the recompile step, etc) for a long time.
I would also point out that "organization based" grouping has an immediate limitation: it is difficult to include a test in more than a group.
I would encourage everyone needing this feature to take a look at TestNG documentation and see how simple things are.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
Re: Test grouping - supported natively in TestNG
by Deborah (Hartmann) Preuss,
Your message is awaiting moderation. Thank you for participating in the discussion.
Yes, I attended Andrew's session on TestNG recently, and was impressed by what TestNG adds. The many-to-many relationship of tests and suites is a good addition. Andrew only mentions it in a sidebar but his blog has a whole section on TestNG, including a comparison of JUnit and TestNG.
Frontline and backline tests
by Slava Imeshev,
Your message is awaiting moderation. Thank you for participating in the discussion.
Good article. This is exactly what we recommend Parabuild users - once the test set grows beyond 5 minutes, split it into a minimum set of tests that run as a part of the continuous integration cycle (frontline) and the set that run on a scheduled basis - hourly, daily (backline).
The frontline test set is essentially a smoke test that ensures that the system is alive after new changes are checked in. The backline test set gives the full picture of the system quality. This may also include full-blown automated unit tests.