Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations The Death of Continuous Integration

The Death of Continuous Integration



Steve Smith compares and contrasts different types of Feature Branching and Trunk Based Development, and explains why Continuous Delivery without Continuous Integration is not working.


Steve Smith is an Agile consultant and Continuous Delivery specialist at Always Agile Consulting Ltd. Steve is a co-author of the Continuous Delivery and DevOps book “Build Quality In“, a co-organizer of the monthly London Continuous Delivery meetup group, a co-organizer of the annual PIPELINE conference, and a regular speaker at conferences such as Agile On The Beach and QCon New York.

About the conference

Agile Tour London conference is a part of the Global Agile Tour. Since 2008, it has become a hugely successful worldwide event spanning 30 countries, creating a world-wide network of agile events open to everyone: from Confirmed Agile Practitioner to Agile Newbie. The 2nd London edition again allowed participants to increase their knowledge about Agile and discuss best practices.

Key Takeaways

  • Many teams now implicitly discard Continuous Integration due to ever-easier Feature Branching and an under-appreciation of Trunk Based Development
  • Continuous Delivery needs a releasable codebase
  • Continuous Integration minimizes integration costs
  • All developers commit to Trunk at least once a day
  • Continuous integration is dying

Show notes

Continuous Delivery

  • 0:58 Continuous Delivery helps companies shorten their time to market, deliver value faster to our customers and grow product revenues
  • 1:36 Continuous Delivery requires a lot of change to people and to technology and to companies and demands to have an always releasable code base
  • 1:55 Continuous Integration ensures that your code base is always integrated, tested and releasable. Continuous Delivery is dependent upon Continuous Integration
  • 3:10 Continuous build is not the same as continuous integrate, so continuous integration is not a tool, it’s a mindset
  • 3:20 Each member of a team commits to Trunk at least once a day

Release feature branching

  • 5:40 With release feature branching, developers would commit their changes to a long-lived shared branch of a source code repository. It was scary because it was complex, risky, unpredictable and would probably cause defects
  • 6:20 Release feature branching example: fictional company account service team are doing release feature branching over a period of some time
  • 6:30 Advantage: Four different epics and the only advantage of this style of work was that if a build failure happened then the other branches could continue unimpeded.
  • 6:50 Problems: 4 different epics merging in at 4 different times and when you do it’s not possible to predict how difficult it’s going to be to actually get Trunk to work after that production release.
  • 7:16 Very little refactoring ever happened and that was simply out of self-preservation.
  • 7:32 Not a lot of collaboration happening: 2 developers working on 2 different branches will have a design diverge in 2 different directions. In this style of work it’s impossible to achieve continuous integration.

Trunk based development

  • 9:00 In the early – mid 2000 Trunk based development was clearly a rival to this style of feature branching
  • 9:10 Trunk based development was and is a version control strategy where developers commit their changes directly onto Trunk in small incremental steps that minimize merge complexity and always preserve a releasable code base.
  • 9:25 Developers are committed to Trunk multiple times a day. Testing happens from Trunk and production releases ideally happen from Trunk.
  • 9:31 You might use a release branch for risk mitigation reasons if a production defect is found.
  • 9:45 Trunk based development is equally applicable to centralized version control systems and distributed version control systems. It’s entirely independent of any particular tool.
  • 10:01 Example: Company Accounts Service – a period of a couple of weeks working on several specific features.
  • 10:16 When 2 features start in proximity to one another in terms of time and region of the codebase with Trunk Based Development there has to be a conversation, you can’t avoid this problem of people treading on one another’s toes.
  • 10:40 What is likely to come out of that conversation, is a decision to conceal one of these features and in the case of a new feature, you would conceal it with something called a 'feature toggle', which is essentially just a runtime configuration parameter.
  • 11:28 With Trunk based development, a build failure is a quite big deal, there is no place to hide. A build failure stops everyone in their tracks.
  • 13:58 Everyone has a shared mental state of the codebase so there can be a very high degree of collaboration.
  • 14:33 Feature toggle and branch by abstraction have operational and business benefits
  • 15:45 Trunk based development is entirely compatible with continuous integration
  • 16:25 Trunk based development requires a big investment in development practices, in automated testing. It requires every member of the team to re-architect, redesign and re-communicate their changes every single day. It can be much harder with a legacy code.
  • 16:55 It feels like you are moving slower, but it means you are spending more time working on something until it’s developer complete.
  • 17:08 If everyone is moving through the code base at the same pace, then you are less likely to have to go into some kind of costly re-architecture or some redesign.
  • 18:12 Continuous Delivery at HP 2008: 400+ engineers on 10m lines of firmware; 10 Release Feature Branches; 1 week build, 6 weeks regression testing; Engineers: 25% product support, 5% innovation
  • 19:00 Trunk based development is entirely compatible with continuous integration

Build Feature Branching

  • 21:50 Build Feature Branching feels like it should be compatible with continuous integration but it isn’t.
  • 22:26 GitHub Flow is entirely synonymous with Feature Branching.
  • 22:30 With Build Feature Branching developers commit to their changes in their individual remote feature branches that were intended to last for a day or so before they emerged back into Trunk for testing and release into production.
  • 22:53 Build Feature Branching development example and explanations
  • 28:18 There are 3 key assumptions that will not hold water for any team for length of time: Assumption #1 - feature branches will always be short-lived. Assumption #2 - reviews will always be timely and assumption #3 - builds will always be well-monitored.
  • 28:33 Assumption #1 - feature branches are always be short-lived. Feature branches last longer than a day and that’s because the only constraint upon feature branch duration is developer discipline. When this happens to one developer on the team, you’ve lost continuous integration.
  • 29:46 Assumption #2 - reviews are always be timely. Reviews last as long as it takes for 2 people who are working on orthogonal tasks to actually come together and share responsibility for a single outcome.
  • 30:50 Reviews might take hours or days because the right person is not available at the right time. To make continuous integration work you have to sacrifice code reviews, if they are not timely enough.
  • 31:18 Assumption #3 - builds are always be well-monitored. Builds will be slower in this style of version control because build feature branching is predisposed towards asynchronous integration.
  • 31:41 James Shore describes synchronous integration as when a developer commits their changes to Trunk and they wait for the build to finish before they move on to the next task and asynchronous integration as when a developer commits their changes to Trunk and they immediately move on to the next task, without waiting for the build to complete.
  • 34:16 Build Feature Branching development is probably incompatible with continuous integration

Continuous Integration is dying

  • 35:26 Why is continuous integration dying? Stackoverflow survey – 16k developers responded to a question about version control and 69% of them said that they use Git as their version control tool of choice
  • 37:14 GitHub – very popular tool. In July 2015 they reported 26m repositories and 10m users
  • 37:57 How the team behind Build Feature Branching, the team behind GitHub are able to use Build Feature Branching to accomplish continuous integration
  • 38:59 Build Feature Branching is rocketing in popularity and that means that continuous integration is dying. Continuous delivery is going to be very difficult to attain
  • 39:25 Trunk based development as a solution
  • 41:00 Q&A

Resources mentioned

Languages, tools and projects mentioned

People mentioned

See more presentations with show notes

DDD and Microservices: At Last, Some Boundaries!

Category Theory for the Working Hacker

Null References: The Billion Dollar Mistake

Simple Made Easy

Recorded at:

Feb 11, 2016