InfoQ

Article

Book Excerpt: Continuous Integration means Continuous Testing

Posted by Paul Duvall, Steve Matyas, Andrew Glover on Aug 05, 2007 01:16 AM

Community
Agile
Topics
Delivering Quality,
Agile Techniques
Tags
JUnit,
TDD,
Testing,
DbUnit,
Continuous Integration,
Selenium,
Fit / Fitnesse,
TestNG

Continuous Integration (CI) is a basic Extreme Programming (XP) practice, but it is also used by teams that would never consider XP for their work. Martin Fowler has pointed out that, at this point, it's become an essential part of any competent software development activity. In their book Continuous Integration: Improving Software Quality and Reducing Risk, Paul Duvall, Steve Matyas and Andrew Glover aim to help teams make this important practice a "non event" on their projects - something that happens unobtrusively and as a matter of course. When successfully implemented, CI ensures that any individual developer's work is only a few hours away from a shared project state and can integrated back into that state in minutes. InfoQ presents a pdf download (14MB) of Chapter 6: Continuous Testing, which presents strategies for writing the different kinds of tests required to ensure system quality.

Testing is a key area for CI improvement, since the bulk of an application's build time is usually applied to running tests. Poorly constructed test suites can cause build times to bog down, at which point teams start to circumvent their own agreed-upon CI practices just to recapture the time they need to code. These shortcuts increase the likelihood of "red" (broken) builds, and can lead to a "broken windows" scenario in which it's impossible to judge code quality because the build runs "red" more often than not, increasing the risk of serious production flaws, and provoking the addition of lengthy pre-release testing that delays deployment.

In this chapter on testing the authors describe the following topics and the relationships between them:

  • Automate unit tests
    Automate your unit tests, preferably with a unit testing framework such as NUnit or JUnit. These unit tests should have no external dependencies such as a file system or database.
  • Automate component tests
    Automate your component tests with unit testing frameworks such as JUnit, NUnit, DbUnit, and NDbUnit if you are using a database. These tests involve more objects and typically take much longer to run than unit tests.
  • Automate system tests
    System tests are longer to run than component tests and usually involve multiple components.
  • Automate functional tests
    Functional tests can be automated using tools like Selenium (for Web applications) and Abbot for GUI applications. Functional tests operate from a user’s perspective and are typically the longest running tests in your automated test suite.
  • Categorize developer tests
    By categorizing your tests into distinct “buckets,” you can run slower running tests (e.g., component) at different intervals than faster running tests (e.g., unit).
  • Run faster tests first
    Run your unit tests prior to running component, system, and functional tests. You can achieve this by categorizing your tests.
  • Write tests for defects
    Increase your code coverage by writing tests based on new defects and ensuring that the defect does not surface again.
  • Make component tests repeatable
    Use database testing frameworks to make certain that the data is a “known state,” which helps make component tests repeatable.
The chapter ends with a series of questions teams may use to evaluate their own CI  test process:
Are you categorizing your automated tests, such as unit tests, component tests, system tests, and functional tests?

Are you configuring your CI system to run each test category with different staged builds?

Are you writing automated unit tests for each defect?

How many asserts are in each of your test cases? Are you limiting each test case to one assert?

Are these tests automatable? Has your project committed automated developer tests to the version control repository?
The chapter is amply illustrated with examples from various testing frameworks, including TestNG, Ruby, DbUnit, StrutsTest, Selenium and JUnit.

Read the InfoQ exclusive excerpt: Chapter 6: Continuous Testing from the book Continuous Integration: Improving Software Quality and Reducing Risk by Paul Duvall with Steve Matyas and Andrew Glover. Have a look at the full table of contents on O'Reilly Safari for an idea of the other topics covered in the book.

4 comments

Reply

Link broken by Michael Hunger Posted Aug 12, 2007 2:59 AM
Link broken by Nikita Salnikov-Tarnovski Posted Aug 14, 2007 7:47 AM
Re: Link broken by Majid Latif Posted Aug 14, 2007 3:27 PM
Re: Link broken by Floyd Marinescu Posted Aug 14, 2007 10:31 PM
  1. Back to top

    Link broken

    Aug 12, 2007 2:59 AM by Michael Hunger

    the download link is broken. Please fix it. Michael

  2. Back to top

    Link broken

    Aug 14, 2007 7:47 AM by Nikita Salnikov-Tarnovski

    +1

  3. Back to top

    Re: Link broken

    Aug 14, 2007 3:27 PM by Majid Latif

    ++

  4. Back to top

    Re: Link broken

    Aug 14, 2007 10:31 PM by Floyd Marinescu

    Many apologies, it has been fixed. So sorry!

Exclusive Content

Business Natural Languages Development in Ruby

Jay Fields presents his concept of Business Natural Languages - a type of Domain Specific Languages geared towards being readable by domain experts.

Distributed Version Control Systems: A Not-So-Quick Guide Through

Adoption and interest for Distributed Version Control Systems is constantly rising. We will introduce the concept of DVCS and have a look at 3 actors in the area: git, Mercurial and Bazaar.

Segundo Velasquez and Agile as Seen Through the Customer's Eyes

Deborah Hartmann interviewed Segundo Velasquez about his experience as customer with an Agile team during the initial phase of software design of a product.

Fine Grained Versioning with ClickOnce

David Cooksey shows how to fine grained versioning to a ClickOnce deployment using an HttpHandler written with ASP.NET, making partial rollouts to a test audience much easier.

Implementing Manual Activities in Windows Workflow

Windows workflow (WF) is an excellent framework for implementing business processes, but lacks support for human activities. This article describes a completely generic approach for changing this.

Markus Voelter about Software Architecture Documentation

In this interview taken during OOPSLA 2007, Markus Voelter talks about the importance of documenting the software architecture, and gives some good and also bad examples on how it could be done.

Voca, UK's largest payment processing engine running Spring

William Soo and Meeraj Kunnumpurath discuss the Voca transaction processing system, architectural challenges and requirements, Voca's Spring/J2EE architecture, and the future SEPA architecture.

Patterns for securing architectures

Security is about trade-offs. Only a few have the expertise to design good security. This talk focuses on Security Patterns, such as Role-based Access Control, Single Access Point, and Front Door.