Book Excerpt: Continuous Integration means Continuous Testing

Posted by Paul Duvall, Steve Matyas, Andrew Glover on Aug 05, 2007 01:16 AM

Community

Agile

Topics

Delivering Quality ,

Agile Techniques

Tags

Continuous Integration ,

TDD ,

Testing ,

JUnit ,

Selenium ,

Fit / Fitnesse ,

DbUnit ,

TestNG

Continuous Integration (CI) is a basic Extreme Programming (XP) practice, but it is also used by teams that would never consider XP for their work. Martin Fowler has pointed out that, at this point, it's become an essential part of any competent software development activity. In their book Continuous Integration: Improving Software Quality and Reducing Risk, Paul Duvall, Steve Matyas and Andrew Glover aim to help teams make this important practice a "non event" on their projects - something that happens unobtrusively and as a matter of course. When successfully implemented, CI ensures that any individual developer's work is only a few hours away from a shared project state and can integrated back into that state in minutes. InfoQ presents a pdf download (14MB) of Chapter 6: Continuous Testing, which presents strategies for writing the different kinds of tests required to ensure system quality.

Testing is a key area for CI improvement, since the bulk of an application's build time is usually applied to running tests. Poorly constructed test suites can cause build times to bog down, at which point teams start to circumvent their own agreed-upon CI practices just to recapture the time they need to code. These shortcuts increase the likelihood of "red" (broken) builds, and can lead to a "broken windows" scenario in which it's impossible to judge code quality because the build runs "red" more often than not, increasing the risk of serious production flaws, and provoking the addition of lengthy pre-release testing that delays deployment.

In this chapter on testing the authors describe the following topics and the relationships between them:

• Automate unit tests
  Automate your unit tests, preferably with a unit testing framework such as NUnit or JUnit. These unit tests should have no external dependencies such as a file system or database.
• Automate component tests
  Automate your component tests with unit testing frameworks such as JUnit, NUnit, DbUnit, and NDbUnit if you are using a database. These tests involve more objects and typically take much longer to run than unit tests.
• Automate system tests
  System tests are longer to run than component tests and usually involve multiple components.
• Automate functional tests
  Functional tests can be automated using tools like Selenium (for Web applications) and Abbot for GUI applications. Functional tests operate from a user’s perspective and are typically the longest running tests in your automated test suite.
• Categorize developer tests
  By categorizing your tests into distinct “buckets,” you can run slower running tests (e.g., component) at different intervals than faster running tests (e.g., unit).
• Run faster tests first
  Run your unit tests prior to running component, system, and functional tests. You can achieve this by categorizing your tests.
• Write tests for defects
  Increase your code coverage by writing tests based on new defects and ensuring that the defect does not surface again.
• Make component tests repeatable
  Use database testing frameworks to make certain that the data is a “known state,” which helps make component tests repeatable.

The chapter ends with a series of questions teams may use to evaluate their own CI  test process:

Are you categorizing your automated tests, such as unit tests, component tests, system tests, and functional tests?

Are you configuring your CI system to run each test category with different staged builds?

Are you writing automated unit tests for each defect?

How many asserts are in each of your test cases? Are you limiting each test case to one assert?

Are these tests automatable? Has your project committed automated developer tests to the version control repository?

The chapter is amply illustrated with examples from various testing frameworks, including TestNG, Ruby, DbUnit, StrutsTest, Selenium and JUnit.

Read the InfoQ exclusive excerpt: Chapter 6: Continuous Testing from the book Continuous Integration: Improving Software Quality and Reducing Risk by Paul Duvall with Steve Matyas and Andrew Glover. Have a look at the full table of contents on O'Reilly Safari for an idea of the other topics covered in the book.

1. Back to top

   Link broken

   Aug 12, 2007 2:59 AM by Michael Hunger

   the download link is broken. Please fix it. Michael

   Reply

2. Back to top

   Link broken

   Aug 14, 2007 7:47 AM by Nikita Salnikov-Tarnovski

   +1

   Reply

3. Back to top

   Re: Link broken

   Aug 14, 2007 3:27 PM by Majid Latif

   ++

   Reply

4. Back to top

   Re: Link broken

   Aug 14, 2007 10:31 PM by Floyd Marinescu

   Many apologies, it has been fixed. So sorry!

   Reply

• Older >