Book Excerpt: Continuous Integration means Continuous Testing

| Posted by Paul Duvall Steve Matyas Andrew Glover on Aug 05, 2007. Estimated reading time: 3 minutes |

Continuous Integration (CI) is a basic Extreme Programming (XP) practice, but it is also used by teams that would never consider XP for their work. Martin Fowler has pointed out that, at this point, it's become an essential part of any competent software development activity. In their book Continuous Integration: Improving Software Quality and Reducing Risk, Paul Duvall, Steve Matyas and Andrew Glover aim to help teams make this important practice a "non event" on their projects - something that happens unobtrusively and as a matter of course. When successfully implemented, CI ensures that any individual developer's work is only a few hours away from a shared project state and can integrated back into that state in minutes. InfoQ presents a pdf download (14MB) of Chapter 6: Continuous Testing, which presents strategies for writing the different kinds of tests required to ensure system quality.

Testing is a key area for CI improvement, since the bulk of an application's build time is usually applied to running tests. Poorly constructed test suites can cause build times to bog down, at which point teams start to circumvent their own agreed-upon CI practices just to recapture the time they need to code. These shortcuts increase the likelihood of "red" (broken) builds, and can lead to a "broken windows" scenario in which it's impossible to judge code quality because the build runs "red" more often than not, increasing the risk of serious production flaws, and provoking the addition of lengthy pre-release testing that delays deployment.

In this chapter on testing the authors describe the following topics and the relationships between them:

  • Automate unit tests
    Automate your unit tests, preferably with a unit testing framework such as NUnit or JUnit. These unit tests should have no external dependencies such as a file system or database.
  • Automate component tests
    Automate your component tests with unit testing frameworks such as JUnit, NUnit, DbUnit, and NDbUnit if you are using a database. These tests involve more objects and typically take much longer to run than unit tests.
  • Automate system tests
    System tests are longer to run than component tests and usually involve multiple components.
  • Automate functional tests
    Functional tests can be automated using tools like Selenium (for Web applications) and Abbot for GUI applications. Functional tests operate from a user’s perspective and are typically the longest running tests in your automated test suite.
  • Categorize developer tests
    By categorizing your tests into distinct “buckets,” you can run slower running tests (e.g., component) at different intervals than faster running tests (e.g., unit).
  • Run faster tests first
    Run your unit tests prior to running component, system, and functional tests. You can achieve this by categorizing your tests.
  • Write tests for defects
    Increase your code coverage by writing tests based on new defects and ensuring that the defect does not surface again.
  • Make component tests repeatable
    Use database testing frameworks to make certain that the data is a “known state,” which helps make component tests repeatable.
The chapter ends with a series of questions teams may use to evaluate their own CI  test process:
Are you categorizing your automated tests, such as unit tests, component tests, system tests, and functional tests?

Are you configuring your CI system to run each test category with different staged builds?

Are you writing automated unit tests for each defect?

How many asserts are in each of your test cases? Are you limiting each test case to one assert?

Are these tests automatable? Has your project committed automated developer tests to the version control repository?
The chapter is amply illustrated with examples from various testing frameworks, including TestNG, Ruby, DbUnit, StrutsTest, Selenium and JUnit.

Read the InfoQ exclusive excerpt: Chapter 6: Continuous Testing from the book Continuous Integration: Improving Software Quality and Reducing Risk by Paul Duvall with Steve Matyas and Andrew Glover. Have a look at the full table of contents on O'Reilly Safari for an idea of the other topics covered in the book.

Rate this Article


Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Link broken by Michael Hunger

the download link is broken. Please fix it.


Link broken by Nikita Salnikov-Tarnovski


Re: Link broken by Majid Latif


Re: Link broken by Floyd Marinescu

Many apologies, it has been fixed. So sorry!

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

4 Discuss