Is Pipelined Continous Integration a Good Idea?
To address this issue, many teams “pipeline” their CI, that is they build in stages. A quick build with the fast tests is run and a preliminary pass-fail is sent to the team, while the more extensive form of the build is done in the background. Slower tests, such as functional/integration/system tests, are performed in later stages. Simon Stewart opines on the virtues of such a solution:
And this is how we end up with a build pipeline. Later stages tend to run slower than earlier stages, but everything is arranged to provide fast feedback. We know that we only have to manually test the builds that make it through the far end of the pipeline, and as we add stages we can add confidence that the application works as expected. If we're smart (quick! Pull the "clever" lever again!) we can automatically deploy the app into increasingly realistic environments and run tests against it as part of the pipeline, which is something we'd never imagine doing from the developer workstation.Not everyone agrees that this is a good idea, however. Julian Simpson of ThoughtWorks calls this the “pipeline of doom”. He suggests that we are skirting the issue by using a pipeline – that of a slow build machine. By pipelining the build, we give false confidence that the integration phase is successful. Developers continue to build on bad code for several iterations, which only exacerbates the problem. As for pushing slower tests back:
The other aspect of the pipeline that I find troubling is this: by not forcing the developers to sit through the functional tests before they check in their code, you prevent their fantastic brains from reflecting on improving them. If people are feeling the pain of running them, they have a good motivation to fix them. Those tests ought to provide the biggest bang for your buck: unless you're careful you could be running poor tests dozens of times a day.So, does your team have pipelined CI? How’s it working for you?
Symptom of a deeper problem
- A large project which should be splitted in smaller modules.
- A wrong test strategy (unit tests not relying on mock objects but on external resources).
- A slow build tool.
A pipeline build is more of a hack in my opinion in most situations.
Re: Symptom of a deeper problem
Pipelining CI moves in the wrong direction. Teams that have focused on making the build faster have more complete and faster feedback.
Pragmatic, not a pipeline of doom
We have the challenging task of reducing the length of the feedback loops in as many places as we can. I think we want to focus on the most valuable areas first, and I believe continuous integration has a very high return on investment. Speeding integration by pipelining or by parallel processing of stages of the integration has helped my team see more quickly when something is wrong.
I believe there are operations which are necessary for delivery to the customer, but long enough duration, or expensive enough, that we choose to stage them, delay them, or pipeline them.
We don't burn a DVD with every build from the continuous integration machine, because we've found that we don't learn a lot from burning a DVD.
We run tests from every build on the continuous integration machine, even though the tests take longer than we'd like, because we continue to learn from the running of the tests.
We continue to seek ways to increase the performance of the tests, but have also continued to add more tests. The team (and the stakeholders) have accepted that increasing number of tests may slow integration, but the slower integration is better than the alternatives.
Re: Pragmatic, not a pipeline of doom
Here's what I've seen when this happens:
1) Functional tests aren't run and build passes.
2) Someone checks in code, passes build, but breaks functional tests (and doesn't know it).
3) 5 others check in code in an (unknown to them) broken build.
4) The next stage finally catches up and breaks - but 6 people have checked in. They are *all* sure it wasn't their code that broke the build. So they expect the others to fix it.
Re: Pragmatic, not a pipeline of doom
In many cases, introducing a continuous integration system into complex system is something that takes a long time. For example, in GigaSpaces it was very challenging for us to automate the process and make sure all unit and functional tests are run on every build. We also needed to invest in parallelism of the test harness to make sure the time it takes to run those is reduced linearly based on the test machines available.
Not an "either-or" - adapt to individual circumstances
Developers "building on bad code for several iterations" isn't going to be fixed by running different builds.. - again indicative of deeper team / management problems.
If you have a unit test build that takes say 3 or 4 minutes, and then a full build that takes say 30 minutes - and you are able to identify who is causing the breaking build in both cases - say by running something like Team Piazza or build-o-matic, and you agree that failed builds should be fixed immediately (which is after all the whole point of the CI exercise) - then that seems a reasonable compromise - fast feedback but still no developer wait-states.
Necessary for large-code-base projects
Earlier we had a (set of) customized CruiseControls that would even try to compile the main source of the different modules against each other (as required by the dependency graph) before we wasted time running any tests. Unfortunately this was a little too expensive to maintain ourselves, and quite difficult to port to maven2 builds.
What I lack in CI products is ...
Re: Not an either-or
I agree with your point about the need to question the approach being used if the developers have a 50 minute wait. I guess the point that I was trying to make was that you need to address your slow build and test issues rather than push them further down the pipeline. All that does is reduce throughput through to the QA team.
Extending CI to do more testing than you'd normally do (low intensity performance testing for example): great. Not addressing the causes of the slow build and trying to fix it with a second stage: not so good.
Re: Necessary for large-code-base projects
It sounds like you have a large amount of dependencies. Could the long-term solution possibly involve reducing the coupling instead of pipe-lining the build?