BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Trunk Based Development as a Cornerstone for Continuous Delivery

Trunk Based Development as a Cornerstone for Continuous Delivery

This item in japanese

Bookmarks

Dave Farley and Jez Humble have been championing the importance of trunk based development since long before their pivotal book on Continuous Delivery was published in 2011. Farley recently wrote about trunk based development and the backlash of objections which he has encountered following his talk on Optimising Continuous Delivery at the PIPELINE conference in March. In support of this, Humble shared his own perspectives in a lengthy Twitter thread which inspected the cultural aspects of trunk based development to understand its relation to the programmer mindset.

Farley described trunk based development as the practice which he "gets the most push back on." Trunk based development encourages teams to work in a fashion where they push incremental changes, at least daily, to a shared and always releasable mainline trunk. In contrast to the practice of using longer-lived feature branches, this facilitates early learning and shared feedback for everyone with a stake in the product being built. "Most teams don't merge their branch until the 'feature' that they are working on is complete," wrote Farley who describes trunk based development as a cornerstone of continuous integration (CI) and continuous delivery (CD). He wrote that such feedback isolation is contrary to the practice of CI:

...if CI is about exposing our changes as frequently as possible, so that we can get great feedback on our ideas, branching, any form of branching, is about isolating change. A branch is, by-design, intended to hide change in one part of the code from other developers. It is antithetical to CI, the clue is in the name "CONTINUOUS INTEGRATION".

Farley described feature branching as often providing a misleading sense of stability, within a dynamically changing code base:

Feature Branching is very nice from the perspective of an individual developer, but sub-optimal from the perspective of a team. We would all like to be able to ignore what everyone else is doing and get on with our work. Unfortunately code isn't like that. Even in very well factored code-bases with beautiful separation-of-concerns and wonderfully loosely-coupled components, some changes affect other parts of the system.

Picking up on this Humble wrote that trunk based development is about "putting the needs of the team above the needs of the individual." He pointed out that effective trunk based development encourages communication and working with small batches:

We're using version control to communicate what we're doing to the rest of the team. To do it regularly enough, we have to work in very small batches. This is antithetical to the way lots of developers like to work: sitting off on their own going down a coding rabbit hole for days before re-emerging.

Humble wrote that CI and its prerequisite of trunk based development are "social and team" activities, creating a "challenge to the mythos of the developer-as-hero":

Thus my hypothesis about why feature branching vs CI / trunk based development is such a hot button issue: we are striking at one of the central beliefs about what it means to be a "good" programmer. There's also the fact that people aren't trained to work in small batches and find it uncomfortable.

Considering the effort which teams can put into testing and seeking feedback loops against branches mismatched with what will be released, Farley wrote:

The definitive point is the testing that happens at the point of merge to trunk. It is only at this point that you can honestly say, "Yes, my change works with everyone else's." Before that, you are hoping that someone else hasn't done something horrid on another branch that breaks your stuff when you merge.

Farley wrote that trunk based development is a core component of being successful with CI and CD:

CI is not a naive approach; it is well-thought out and very widely practised in some of the most successful companies in the world. Trunk-based development is a core practice to CI and CD, it really is very difficult to achieve all of the benefits of CI or CD in the absence of Trunk-based development.

Farley has refuted objections to trunk based development, citing the State of DevOps Report and his own personal experience of having succeeded with the practice for several decades, across varied team sizes, programming languages and scales.

Humble recently co-authored the book Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, which took on the challenge of empiricising, collating and statistically quantifying the impact of companies practising continuous delivery. In his twitter thread about Farley's post, Humble tweeted:

...we did some research on working off trunk in the State of DevOps Report, which is also in our new book, Accelerate. The results are clear. Teams which work off trunk or branches which last less than a day have significantly higher performance.

In his forward for Accelerate, Martin Fowler summarised the benefits of working with these practices:

They describe how effective IT delivery organizations take about an hour to get code from committed-to-mainline to running-in-production, a journey lesser organizations take months to do. They thus update their software many times a day, instead of once every few months, increasing their ability to use software to explore the market, respond to events, and release features faster than their competition. This huge increase in responsiveness does not come at a cost in stability, since these organizations find their updates cause failures at a fraction of the rate of their less-performing peers, and are usually fixed within the hour."

 

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • video with Dave talking about not branching

    by Manuel Pais,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Dave recently gave a talk where he talks about why not to branch:

    youtu.be/a8_K5TIH-6I?t=1128

  • Corrections & Clarifications

    by Brad Appleton,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    In the article/blog, Farley does not indicate that he gets the most-pushback on trunk-based-development. Instead, Farley makes a much stronger/broader statement against any and all forms of branching, saying "Don't branch" x# (meaning, never), and then states any form of branching is "antithetical" to Continuous Integration.

    It is this hardline statement that Farley claims he gets the most pushback on, because he is basically saying "no branching!".

    In this case, the pushback is well warranted because several of Farley's reasons why are in fact invalid oversimplifications. First and foremost is the assumption that all branches hide changes. This is incorrect: a branch does in fact isolate changes, but they are very much visible/transparent in the central repo, which is more than can be said for an in-progress change that exists only in the developer's local repo/working-tree (prior to integrating).

    The other false claim is that the branch itself, by its very existence, is the cause of delayed integration. As if it somehow prevents the developer from integrating to main-trunk (any more so than the same work-in-progress if it existed solely in the developer's local repo/working-tree and not also in the central-repo). But this delay is caused by not merging sooner (and more frequently), regardless of whether the work is only in the local repo/workspace or also on a branch.

    Humble get's it right when he says the problem is lack of breaking-down the work into smaller chunks that are integrated more frequently (i.e. continuously).

    Perhaps the most underlying fundamentally false premise is that when a (feature/topic/story/task) branch is created, that it is only ever merged to main-trunk at the very end after all the desired work/functionality is complete. This is certainly a common anti-pattern in use, but it does not prevent or preclude frequent/continuous integration from the branch to the trunk throughout the lifetime of the branch.

    Unfortunately, this is exactly the assumption that is made in the 2018 "Accelerate" book (and the 2017 + 2018 State of DevOps Reports) that Farley refers to as providing conclusive data branches delay integration (the assumption that the duration/lifetime of the branch is therefore what defines how frequently the work is integrated to trunk -- instead of actually looking at the number+frequency of commits to trunk from the branch in question).

    This false premise is what leads to misinterpreting the resulting data (branch-lifetime) to conclude that it is more effective to create+delete many branches, each living no more than 24hrs, then it is to create a single topic-branch that might live upto 1week but still regularly merges to trunk at least daily (or more frequently) during its lifetime. Yet a simple experiment shows that it is clearly more overhead to create+delete 5+ "mayfly branches" during the same period of time than to create a a single branch that still merges comparably frequently (e.g., even every 8hrs +5min instead of every 8hrs).

    The real lesson here is not "No Branching" but to instead "*Know* Branching" - in order to get the optimal result.

  • Re: Corrections & Clarifications

    by Franklin Nwa,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Farley's argument is totally sound. Why create a branch and then merge to trunk when you can just make the change directly in trunk, and gaining the benefits that come with doing so ?

  • Re: Corrections & Clarifications

    by Brad Appleton,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hi Franklin,
    The statement in your comment is different from Farley's hardline argument+claims (the one he says gets the most pushback).

    That said, the question you asked is the case of a single (very small) change, and what happens in the time between when the change is first started and the time when it is merged to trunk.

    Lets assume the two cases you mentioned in your question ...

    Case 1: No branch is created in the central repository. In fact, no commits are made to the central repository prior to merging to main-trunk. All in progress modifications are in the developer's working-tree (which, if using a DVCS like Git is a local repository). There might be commits of in-progress work in the local repository, but we don't commit any changes to trunk in the central repository if we know its not working+tested yet and would "break the build"

    Case 2: Let's assume we *do* make a development branch in the central repository. We still make changes in the developer's working-tree (or local repo, if using a DVCS). Some commits of work-in-progress to the local repo might get "synced" to the dev-branc in the central repo (but not yet merged to trunk if its not working+tested yet).

    Now let's look at Farley's claims about the differences between these two cases, and see what is true and what isnt.

    CLAIM #1: Case 2 is somehow hiding the change and its current status from the rest of the team, from the central repository, and from the main-trunk (or at least much more so than Case 1)

    This is obviously false.
    - In Case 1, *nothing* is visible in the central repository, because all in-progress changes & commits exist solely in the developers working-tree (local repo). You can't even tell from the central-repo whether or not the change has started, much less in-progress.

    - In case 2, the dev-branch is visible in the central-repo. *And* if it is named at all meaningfully, then it gives a clue that not only has work-started, but what the change is for (e.g., a backlog-item) and at once both declares (reveals) its intent/purpose, PLUS, if there are any commits on the branch, another developer can see/investigate the result, diff/compare, and more.

    CLAIM 2: there is a significant delay between when code is merged-to-trunk in case 1 vs case 2 (where *significant* means on the order of days, or hours, or at least a few minutes). And *somehow* the existence of the branch automatically encourages/motivates such a delay.

    This claim doesn't hold water either. the reason code isn't yet merged to trunk is because it either is not yet working+tested, and/or there is a (rational or irrational) fear of merging (e.g., that it will break-the-build, or be very painful). The existence (or absence) of the dev-branch does *not* make the change any easier or harder to breakdown into smaller, testable chunks; does *not* make it more or less likely to break the build, nor make it easier or harder to merge. Either way, there are unmerged changes (of the pretty much the same size/complexity)

    All of that depends solely on other factors having to do with the amount/complexity of modifications made so far, and with the size+number of other changes merged to trunk since our dev-change first began (REGARDLESS of whether or not a dev-branch is used. There are still unmerged changes sitting in the developers working-tree (or local repo). The only difference is that in once case, some of it is visible in the central repo.

    NOW, in both cases, let's assume changes have been locally built+tested and are ready to be merged to main-trunk to trigger the CI process. (Let's assume any new/updated files were already staged/added to the local working-tree/repository).

    - case 1: the change is merged directly to main-trunk and the CI process is triggered.
    - case 2: the change is first "synced" to the dev-branch (which takes *seconds*, certainly not minutes, nor even a whole minute. Then the actual merge happens to the main-trunk (triggering the rest of the CI process).

    You could argue that for case2 you have to manually enter two separate consecutive commands, except that's not really true because it is trivial to trigger them from a single action, whither via an IDE client (like Eclipse+egit) and/or something in my vcs-config (e.g., gitconfig), or even use a separate "alias" for a checkpoint/sync commit, versus a "merge-to-trunk" commit.

    So, at best, any so called "delay" here is a matter of seconds and not even remotely close to the "significant" range (certainly not days or hours, nor even minutes).

    PLUS - in case2 there *is* an added benefit that doesn't exist in case1. That of transparency + visibility to the rest of the team (in fact it is the opposite of Farley's claim. It is not case2 that "hides" current status/progress from the team, it is the changes that exist ONLY in the local working-tree/repo that are hidden from the rest of the team.

    So the tradeoff is along the lines of hidden changes/status/progress for the lifetime of the change, versus visibly transparent, intention-revealing status/progress for the lifetime of the change, at the expense of a few extra seconds each to create+sync the dev-branch.

  • Re: Corrections & Clarifications

    by Brad Appleton,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    After a very long response from a few years ago, Ive returned to give a more concise one ...

    *QUESTION*:: "Why create a branch and then merge to trunk when you can just make the change directly in trunk, and gain the benefits that come with doing so?"

    *SHORT ANSWER*:: Because after making the change, you'll "merge [directly] to trunk" *regardless* of whether or not you created a branch (and still gain the benefits of doing so).

    - Using a branch provides transparency - making WIP (work-in-progress) visible *and* revealing its intention!

    - NOT Using a branch *hides* BOTH (WIP + intent), until AFTER youve committed to the central repository!

    Aside from that ... the BIGGEST benefit comes from frequently merging to main-trunk, *throughout* the lifetime of the development task (*regardless* of whether or not you created a branch).

    ----------------------------------

    *LONGER EXPLANATION*::
    If you made the change in your own local workspace/repo - then you must (eventually) merge to trunk. That has nothing to do with using a branch (or not), it's because you made concurrent/parallel changes in a private+isolated area.

    The only real difference is whether or not your "in-progress" work is visible (e.g., read-only) in the central repository. If you used a branch for that - it is visible; if not - no one has any clue about it until AFTER you merge to main-trunk.

    The length/lifetime of the branch is irrelevant - what matters is the lifetime of the development-task *AND* that you merge to main-trunk as frequently as possible throughout the lifetime of the development-task.

    The problem is the most people are loathe to merge until the ENTIRE task is completed! The false assumption/assertion is that using a branch makes this more likely (or more tempting).
    - This has NEVER been the case.
    - The mental block has to do with merging mid-TASK/story/feature
    - They are afraid to merge INCOMPLETE work (even if its in a working+tested state)
    Using a branch for that work just makes it more transparent.

    Learning to breakdown a seemingly "elementary" unit of work into smaller working "inch/pebble" milestones (like with test-first, or stepwise refinement), and then merging at each milestone along the way -- is what makes continuous integration happen (and failing to do so impedes that).

    AGAIN - this is regardless of whether the "unitary" work-item (task/story/feature) is hosted in a workspace WITH a branch (visible+transparent), or withOUT (hidden)!

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT