Is Pipelined Continous Integration a Good Idea?

Posted by Amr Elssamadisy on Sep 28, 2007

Sections

Process & Practices

Topics

Agile Techniques ,

Agile in the Enterprise ,

Agile

Tags

Continuous Integration

One of the well-known practices of Agile development is Continuous Integration, which entails team members integrating their code regularly into the baseline and running all unit and system tests.  In most teams a CI server is used to do this quickly and automatically when code is checked into the baseline.  This usually works well in the beginning of a project, but sometimes, when the team and/or code-base get large, the CI server starts to slow down.  The cycle between builds grows and the feedback degrades – a build may take an hour or more to respond with a pass/fail, and by that time several people may have checked in their code into an already broken build.  

To address this issue, many teams “pipeline” their CI, that is they build in stages.  A quick build with the fast tests is run and a preliminary pass-fail is sent to the team, while the more extensive form of the build is done in the background.  Slower tests, such as functional/integration/system tests, are performed in later stages.  Simon Stewart opines on the virtues of such a solution:

And this is how we end up with a build pipeline. Later stages tend to run slower than earlier stages, but everything is arranged to provide fast feedback. We know that we only have to manually test the builds that make it through the far end of the pipeline, and as we add stages we can add confidence that the application works as expected. If we're smart (quick! Pull the "clever" lever again!) we can automatically deploy the app into increasingly realistic environments and run tests against it as part of the pipeline, which is something we'd never imagine doing from the developer workstation.

Not everyone agrees that this is a good idea, however. Julian Simpson of ThoughtWorks calls this the “pipeline of doom”. He suggests that we are skirting the issue by using a pipeline – that of a slow build machine. By pipelining the build, we give false confidence that the integration phase is successful. Developers continue to build on bad code for several iterations, which only exacerbates the problem. As for pushing slower tests back:

The other aspect of the pipeline that I find troubling is this: by not forcing the developers to sit through the functional tests before they check in their code, you prevent their fantastic brains from reflecting on improving them. If people are feeling the pain of running them, they have a good motivation to fix them. Those tests ought to provide the biggest bang for your buck: unless you're careful you could be running poor tests dozens of times a day.

So, does your team have pipelined CI? How’s it working for you?

1. Back to top

   Symptom of a deeper problem

   by Alexandre Poitras

   A slow build is usually the reflect of a deeper problem :
   - A large project which should be splitted in smaller modules.
   - A wrong test strategy (unit tests not relying on mock objects but on external resources).
   - A slow build tool.
   - ...
   
   A pipeline build is more of a hack in my opinion in most situations.

   Reply

2. Back to top

   Re: Symptom of a deeper problem

   by Amr Elssamadisy

   I tend to agree. Unfortunately many teams handle painful steps by doing less of that step. It usually causes more pain in the future, but somehow we miss the connection. (Integration at the end of a project using a traditional software process quickly comes to mind.)
   
   Pipelining CI moves in the wrong direction. Teams that have focused on making the build faster have more complete and faster feedback.

   Reply

3. Back to top

   Pragmatic, not a pipeline of doom

   by Mark Waite

   I think Julian is wrong to call pipelined continuous integration a "pipeline of doom". I believe there are many stages in the pipeline from user story to customer accepted and installed product.
   
   We have the challenging task of reducing the length of the feedback loops in as many places as we can. I think we want to focus on the most valuable areas first, and I believe continuous integration has a very high return on investment. Speeding integration by pipelining or by parallel processing of stages of the integration has helped my team see more quickly when something is wrong.
   
   I believe there are operations which are necessary for delivery to the customer, but long enough duration, or expensive enough, that we choose to stage them, delay them, or pipeline them.
   
   We don't burn a DVD with every build from the continuous integration machine, because we've found that we don't learn a lot from burning a DVD.
   
   We run tests from every build on the continuous integration machine, even though the tests take longer than we'd like, because we continue to learn from the running of the tests.
   
   We continue to seek ways to increase the performance of the tests, but have also continued to add more tests. The team (and the stakeholders) have accepted that increasing number of tests may slow integration, but the slower integration is better than the alternatives.

   Reply

4. Back to top

   Re: Pragmatic, not a pipeline of doom

   by Amr Elssamadisy

   Stages such as what? Almost everyone will agree to the DVD burning stage - but not running functional/system tests?!
   
   Here's what I've seen when this happens:
   1) Functional tests aren't run and build passes.
   2) Someone checks in code, passes build, but breaks functional tests (and doesn't know it).
   3) 5 others check in code in an (unknown to them) broken build.
   4) The next stage finally catches up and breaks - but 6 people have checked in. They are *all* sure it wasn't their code that broke the build. So they expect the others to fix it.

   Reply

5. Back to top

   Re: Pragmatic, not a pipeline of doom

   by Guy Nirpaz

   I find this discussion very valuable. The basic question that is being asked, is how much pragmatism team afford in their processes?
   In many cases, introducing a continuous integration system into complex system is something that takes a long time. For example, in GigaSpaces it was very challenging for us to automate the process and make sure all unit and functional tests are run on every build. We also needed to invest in parallelism of the test harness to make sure the time it takes to run those is reduced linearly based on the test machines available.

   Reply

6. Back to top

   Not an "either-or" - adapt to individual circumstances

   by James Richardson

   Quite ovbiously the answer is dependent on the particular implementation. Forcing every developer to wait for 50 minutes to run a test suite locally before checking in is a fantastic waste of time. Anybody doing this should immediately question what they are doing. Its indicative of some serious problems with the development infrastructure - test design, application design, poor hardware, poor network or poor team setup.
   Developers "building on bad code for several iterations" isn't going to be fixed by running different builds.. - again indicative of deeper team / management problems.
   If you have a unit test build that takes say 3 or 4 minutes, and then a full build that takes say 30 minutes - and you are able to identify who is causing the breaking build in both cases - say by running something like Team Piazza or build-o-matic, and you agree that failed builds should be fixed immediately (which is after all the whole point of the CI exercise) - then that seems a reasonable compromise - fast feedback but still no developer wait-states.

   Reply

7. Back to top

   Necessary for large-code-base projects

   by Eirik Maus

   we have several projects with many shared modules. If everything (200 modules or so) was to run in a single CI, it would take the entire day to build through it all.
   
   Earlier we had a (set of) customized CruiseControls that would even try to compile the main source of the different modules against each other (as required by the dependency graph) before we wasted time running any tests. Unfortunately this was a little too expensive to maintain ourselves, and quite difficult to port to maven2 builds.

   Reply

8. Back to top

   What I lack in CI products is ...

   by Eirik Maus

   ... a simple way to monitor change in random URLs, so that the different CI instances in the pipeline (or, rather, build graph) more easily can get triggered by each other. Now we often have to push the "build now" button in order to have the changes in one project's CI propate through all dependent modules in different CI instances.

   Reply

9. Back to top

   Re: Not an either-or

   by Julian Simpson

   James,
   
   I agree with your point about the need to question the approach being used if the developers have a 50 minute wait. I guess the point that I was trying to make was that you need to address your slow build and test issues rather than push them further down the pipeline. All that does is reduce throughput through to the QA team.
   
   Extending CI to do more testing than you'd normally do (low intensity performance testing for example): great. Not addressing the causes of the slow build and trying to fix it with a second stage: not so good.

   Reply

10. Back to top

    Re: Necessary for large-code-base projects

    by Amr Elssamadisy

    Please tell us more about your project... C/C++, .NET, Java? The 200 modules - are they dynamically linked? Are they all one project or several projects in an enterprise?
    
    It sounds like you have a large amount of dependencies. Could the long-term solution possibly involve reducing the coupling instead of pipe-lining the build?

    Reply

11. Back to top

    Re: What I lack in CI products is ...

    by Matt Doar

    I think that those dependencies really belong in the build system, and then the target called by the CI can change instead.

    Reply

• Older >