More Feature Branching Means Less Continuous Integration

my own experience is very similar, though I couldn't explain why nearly as concisely

Vendors and teams often silently redefine Continuous Integration to mean something different in context of feature branch workflow. Something like automated builds or pre-merge validations, etc.
I prefer using another name for this usage pattern, e.g. "Jenkins" instead of "CI server".

Thanks Peter!

Thanks Pavel, I agree with you. I always say "build server".

I'm not sure what is the conflict between continuous integration and any sort of branches. As long as people regularly push these branches, have Jenkins build "*", or all branches.

If you want to fancy it up a notch, you can have even have Jenkins automatically merge these WIP-branches with the main branch before building: stackoverflow.com/questions/17696653/how-to-aut...

If people let their branches go too long without merging back, you've got bigger problems than your branching strategy.

Hi Thomas

I went into some detail above, some more in my linked blog posts, and even more in my talk - but in short...

Continuous Integration means *every* member of a team committing to master *at least* once a day. If that can be accomplished with a version control strategy that involves feature branching, all well and good. However, I believe it is highly unlikely that will be the case for the majority of teams.

Continuous Integration is orthogonal to Jenkins, TeamCity... they are all just build servers. Yes, you can build branches automatically. Yes, you can merge automatically.... but they all just tools. The important thing is the practice of Continuous Integration, and the use of tools that encourage rather than discourage the practice.

Cheers

Steve

How about practicing merging the trunk/master branch to feature branch on daily basis and doing the CI on every feature branch. Once successful, merge it to master and then run the CI and other automates component testing.

Hi Amardeep

You are talking about 2 separate things:

1. merging from trunk to a branch on a regular basis is known as Promiscuous Merging and is a good idea
2. building branches on a build server is known as Continuous Build or Continuous Isolation, and is an OK idea

Neither are related to, or as valuable as, Continuous Integration - which is everyone committing to trunk on at least a daily basis. Building branches is nowhere near as useful as it only gives you feedback on your changes, not the entire team. For more see www.youtube.com/watch?v=q-dPdpxIcZQ and www.alwaysagileconsulting.com/version-control-s...

@Thomas: I think the conflict comes from the tension that if you have branches, the point of them is to separate some commits from all the others. Whereas for continuous integration, the point is to combine everybody's commits together.

If a developer is going to push changes from their branch into an master or integration branch, then first they will have to pull all changes from the integration/master branch. If they are doing this regularly, then what purpose is the branch serving? Isn't it functionally identical to pulling from then pushing to github master?

@Jonathan: A branch can be very practical, and it doesn't have to lead to more seldom integration with the central master. The purpose it serves is to enable the developers being able to commit (and push) as early as they like, and to group a set of commits as belonging together, annotated by the merge-commit that tie them into master.

@Thomas. Thanks for the reply.

I don't understand yet how a branch "enables developers to commit (and push) as early a they like". If I don't use a branch, then I can still do that. Sorry if I'm being dumb here, I'm not deliberately trying to misinterpret.

The idea that a branch can be useful because the resulting merge commit might present useful information is a new one to me. I haven't seen this used, in practice. I'll have to ponder on it a little.

I guess my position is that if the branch *does* lead to more seldom integration, then it's causing a problem. And if it *doesn't*, then you are pulling and pushing from integration all the time, so why can't you just do that directly, without the branch as an intermediary.

@Jonathan: I had another epiphany today about this. Perhaps it conveys the point I'm trying to make better.

We agree that periodically we want to integrate our changes, and it is good to keep this period short. At some point while coding, we think "now I'm happy with my work, and I want to integrate it", and historically in SVN-times, that would mean creating a single commit of all the local changes and pushing it.

Now these days, Git gives us a bit more flexibility. First of all, we can make multiple commits before we integrate, and this is nice because we can comfortably split our refactorings from functional changes in separate commits, and generally maintain a higher level of commit hygiene (a commit should change one thing, good commit messages, etc). It gives us more chances to review our commits before integrating them.

Now, I still haven't said anything about branches, but when you think about it, every time we integrate, it is merging our local branch with the remote one. Now you could argue that I'm just being technical, that's not the kind of branches you're talking about here. But if you were not working by yourself on these commits, but perhaps remote pairing with another developer, you could have interactions of commits being suggested, ammended and discussed between you before you choose to integrate it with the master branch, thereby committing it to the permanent history of the project.

I feel I'm not explaining this as clearly as it could be done (it is a bit late in the eve), but trying to reword it: The history that does get integrated into the main branch should be of a higher quality than the first attempted commits that are made on local and remote work-in-progress branches. Shying away from the use of such branches can lead to premature integration, leading to poor commit hygiene and history.

It also just occurred to me, that avoiding branches can reduce the tendency to share and collaborate on code before it gets integrated. If I work without branches, always pushing to master, my unit of work is the sum of local changes I have on my workstation, and the only collaboration on that will have to be done pre-commit, typically by pair programming.

And just to bring it into continuous integration, or rather the continuous building/testing part of it - it's possibly that anyone working with a single branch/early integration policy are likely to have their code tested later and less than those working with branches, as their work will fester for a longer period of time in the local changes before they deem it ready to be pushed (commit anxiety).

I hope some of that makes sense :)

If you want to find an example of merge commits representing larger units of work, look no further than Git itself: github.com/git/git/commits/master

Hi Jonathan

I entirely agree

Steve

Hi Jonathan and Thomas

(InfoQ truncated my comment edit so I am reposting)

I have to say - unsurprisingly - that I largely agree with Jonathan and disagree with Thomas. I would like to pick up on a couple of Thomas' points.

Re "we agree that periodically we want to integrate our changes" - we do not want to periodically integrate, we want to integrate as often as possible. Mike Roberts once said "continuous means more often than you think" and I strongly agree with that sentiment. The more often everyone commits to trunk, the smaller the batch size and that unlocks many benefits.

Re "this is nice because we can comfortably split our refactorings from functional changes in separate commits, and generally maintain a higher level of commit hygiene" - commit hygiene is important, but with Trunk Based Development you *have* to have good commit hygiene. Just as Trunk Based Development encourages people to collaborate, to communicate, and to design their work well it also encourages small commits and good commit messages.

Re "shying away from the use of such branches can lead to premature integration, leading to poor commit hygiene and history" - as outlined above I disagree with that. During 7 years of Trunk Based Development at 3 different companies I did not see that. I think poor commit hygiene is more to do with bad developer practices, and I would argue working on a branch can facilitate such practies.

Re "avoiding branches can reduce the tendency to share and collaborate on code before it gets integrated. If I work without branches, always pushing to master, my unit of work is the sum of local changes I have on my workstation, and the only collaboration on that will have to be done pre-commit, typically by pair programming" - we must have a different definition of sharing and collaboration! As I outlined in the original blog series, the above interview, and in a talk I'm doing at the moment I do not believe branching encourages collaboration. It *can* happen with Build Feature Branching e.g. GitHub Flow but it is optional, whereas with Trunk Based Development it is mandatory. I would also say that pre-commit collaboration is preferential to any kind of post-commit collaboration as it avoids rework. n developers on n branches will diverge in n directions over time, due to Conway's Law

Cheers

Steve

Hey @Thomas.

Thanks for taking the time to lay out your thoughts so carefully. I've read it very slowly and carefully, and plan to do so once again later today over a cup of tea, to make sure I'm absorbing what you're trying to say. (Always be open to being shown that I'm wrong, is my motto. :-)

I can see that we share a lot of high-level goals, such as improving collaboration, and high standards of commit hygiene.

However, I don't agree that the best way to achieve high levels of hygiene is to commit locally, then review and fix up before merging. I think it's to make sure the initial commits are correct up-front, using TDD and (yep, as you predicted) pairing.

In your opening para, where you say "At some point while coding" - my opinion is that optimally, this point is as early as possible, i.e. after every commit. This makes the content of every developer's branch end up being identical to the content that would be on master, if every developer was working from it.

I don't agree that branches in general increase the tendency for developers to collaborate. While branches do make it *possible* for developers to see and review each other's work before it gets merged, on the flipside, they also make it *possible* for developers to create things in isolation from each other - obviously this is in fact the whole point of a branch in the first place.

Meanwhile, without branches, every developer is *forced* to merge with (and pass tests with) every other developer's changes. If dev A creates a new utility function, then dev B can start using it (and talking about it, fixing it, improving it) as soon as it is committed. IMHO, this actually creates and enhances opportunities for communication and collaboration.

> The more often everyone commits to trunk, the smaller the batch size and that unlocks many benefits.

I agree on this, but we obviously we should avoid breaking the build, and polluting the history of the project with unclear commits, and series of commits. How many times haven't we all pushed to master, only to realize that we forgot to squash a commit, or left a misleading note in the commit message? Have you ever been annoyed to track the development of one feature across 5 commits intermingled with everything else that was going on that the same time?

> .. with Trunk Based Development you *have* to have good commit hygiene.

I would say you *need* to, and I would also argue that trunk-based development with no option to work with feature branches makes it harder to achieve this.

> I would also say that pre-commit collaboration is preferential to any kind of post-commit collaboration as it avoids rework

I think this will culminate in a religious discussion of those who prefer focusing on review *before* commits are integrated, and those who do not. Our personal anecdotes will add little to something so culturally and context specific as this. I think my points will resonate heavier with the distributed teams in the "age of GitHub" and open source projects with very high quality thresholds (Git, Chromium, kernel, etc), while yours will do so among co-located, agile teams and startups. My points should shine in critical libraries and products used by many, while yours will be more popular in intensive projects with business deadlines. Many companies deal with multiple sets of these factors, so they should figure out when to pick which strategy.

I can only speculate that over time, organization will move to having smaller and smaller modules maintained by fewer people (microservices, anyone?), with patches being contributed by a much larger set of either colleagues or even open-source contributors. We see this a lot already, perhaps especially in the Node ecosystem, where nearly all of the runtime code is open-source modules, with just a little being actual application specific. In this future world, I can imagine more review, more code reading - and using - than writing. And with it, the contribution threshold will gradually grow higher, making most commits mere suggestions that may or may not be integrated to the master branch, at the maintainer(s) discretion.

Just to add an anecdote of my own: I actually do work in my day-job with a mostly trunk-based strategy, and this works pretty good for our kind of team, in our particular business. We did have one colleage leave a year back, and he went to work for a company, a small startup at the time, where they review all changes before merging to master (think Gerrit). He said since that he would never go back to the old pushing straight to master way, and insists that productivity and cross-team productivity is much stronger as a result of this branch-review-heavy process.

@Jonathan Please see my last reply to Steve as I think that I address some of your thoughts there as well. We agree on many points, up to the stance of whether code should be isolated from the master branch until it has been reviewed appropriately.

My argument is that your approach only allows isolation as long as the code remains uncommitted/pushed (and therefore must be reviewed pre-commit with pairing, etc), and that my approach *allows* review, while not necessarily increasing the period of integration if the situation allows it (i.e. simple, small changes can be integrated quickly without much review).

And as I wrote in my reply to Steve, it's hard to say which of these strategies will be put to best use by which teams, but I think the industry generally will move towards "my" side of the argument over time, as proficiency with tools like Git, Jenkins and Gerrit increases.

Hi Thomas

Thanks for your feedback. You make your point well and I agree with at least some of what you say, let's agree to disagree.

Cheers

Steve

Thomas, can you elaborate how a trunk based strategy increase productivity?

"productivity and cross-team productivity is much stronger as a result of this branch-review-heavy process"

I'm trying to understand what it is that makes teams more productive when working on a trunk in stead of pushing changes directly into the main branch.

@BenLinders

Hi Amardeep,

If I understand your suggestion then it would be an additional layer of "continuous" (daily) merge and build that you would like to add. Benefit would be that possible merge conflicts will pop up earlier. Is that why you would do this?

Also, how does this work when there are many feature branches? Conflicts between feature branches won't pop up until they are merged back to the main branch, right?

@BenLinders

Hi Ben,

I use the terms trunk, master, mainline or main branch interchangeably.

I have written about the advantages of working on, and reviewing branches before merging them to the main branch many years ago: blog.tfnico.com/2010/08/why-distributed-revisio... - at the time it was a bit of a SVN vs Git argument, but this part relates especially well to our discussion here:

You might argue that Subversion users can keeping their changes locally in the workspace, but there's a major constraint: They can't collaborate on their feature. This means missing out on a lot of co-operation, creative process, ideas, code-review and refactoring before the feature ends up in the mainline.

Hi Thomas,

I'm unsure if you've answered my question, which might be because I didn't formulate it well enough.

I fully support that people work together in teams, which means that they will share stuff to get feedback fast and frequently. They can use code reviews, walkthroughs, pair programming and tools to improve the quality of their code.

What I don't understand is how this is related to a branching strategy. Whether you use feature branches or commit directly to the main branch in my opinion should not impact collaboration. I mean, if a programmers wants to have feedback, (s)he doesn't have to commit to get it? You can ask a team member to look at you code, work in pairs, etc, without checking in code.

And even if you would commit code before doing a code review (bit strange to commit before pair programming I guess), how does it make a difference if you commit to a feature branch or to a main branch?

Maybe I'm overlooking something?

@BenLinders

Hi Thomas,

Reading the blog below the second link triggers me:

"There are many tools that will help you with code reviewing, and I can't give you The Perfect Tool For The Job. What currently works well for many people is using the approach with Git where you send a pull request. Most Git-tools (such as Github and Stash) have excellent interfaces for going through the diff between a feature/bugfix branch and the main branch. They allow you to add comments with every line of code, so that the author of the code knows exactly what you're commenting on and is able to fix the right code easily. Similar tools also exist for many of the other version control systems out there."

This blog post suggests to use a tool (it mentions Git, but I guess it can be any tool) to ask for a code review. Maybe I'm old fashioned, but if I want feedback from somebody who's in the same room with me, I'll walk over and ask him.

Of course it will be different when working in remote teams, but even then I would expect that there should be easier ways to ask for feedback than using a CM tool.

What are your experiences with this?

@BenLinders

Hi Ben,

Whether you use feature branches or commit directly to the main branch in my opinion should not impact collaboration. I mean, if a programmers wants to have feedback, (s)he doesn't have to commit to get it? You can ask a team member to look at you code, work in pairs, etc, without checking in code.

And even if you would commit code before doing a code review (bit strange to commit before pair programming I guess), how does it make a difference if you commit to a feature branch or to a main branch?

Hi Thomas,

Thanks for your extensive answer. This is a useful discussion, much appreciated!

I agree that when working remote tools can support the communication, trigger people that you want feedback, and help to work time and place dispersed. When teams are collocated I prefer that people walk over and communicate directly as much as possible, but in the end it is op to people to find out what is most effective.

Once code is committed you can run builds which can give an extra check.

So I think we are aligned on this, right?

@BenLinders

This blog post suggests to use a tool (it mentions Git, but I guess it can be any tool) to ask for a code review. Maybe I'm old fashioned, but if I want feedback from somebody who's in the same room with me, I'll walk over and ask him.

Of course it will be different when working in remote teams, but even then I would expect that there should be easier ways to ask for feedback than using a CM tool.

What are your experiences with this?

• You are incurring a cost on the colleague in question, interrupting and forcing a context-switch on her (i.e. pull-based, or asynchronous review would not destroy her concentration).


• You have not formulated and created a commit (or commits) - so you're leaving that out of being reviewed.


• You're increasing the chance for "works-on-my-machine!" mistakes by having it reviewed on your own environment.


• You're browsing and reviewing your changes in *your* environment, with your editor/IDE preferences, screen setup, your keyboard & mouse control, etc. This makes the review very biased towards the way you navigate the code, giving the reviewer less control and power in the review.

So I think we are aligned on this, right?

Hi Thomas,

Thanks for your extensive motivation.

My experiences are different. Although people can have different setups of their machines, screen, and even use different tools, I've seen people who adjusted to that quite easily and quickly. Sitting together at one workstation usually wasn't a problem.

I recognize the context switch problem, if people are in flow you want them to keep working. Solution would be to agree upon a time frame for review/feedback. It would be instant then, but can still be quick. Direct communication also enhances the quality of the feedback (as you also mentioned in your answer).

I don't understand what you mean with "You have not formulated and created a commit (or commits) - so you're leaving that out of being reviewed". What would a commit comprise, and what can be the benefit of reviewing it?

@BenLinders

What would a commit comprise, and what can be the benefit of reviewing it?

Having multiple reviewers on a workstation can be difficult. I remember doing many code review meetings with one client where attendants printed the code and made remarks on paper. It sounds very inefficient but the group discussions often made it worthwhile to do it. It was also a great way to pull in and clarify coding guidelines when the code review revealed that they weren't understood.

You mentioned that "committing to the main branch as chiseling something into stone" which "cannot be undone. What about doing a roll back? "Shouldn't be to hard to do with modern version management tools.

@BenLinders

Clear, thanks Thomas. This sounds indeed valuable, I've seen many cases where people forgot specific files when implementing a change, which then causes the build to fail.

I usually advice people to make a list of impacted files when working on a ticket or task, and check this it a colleague if they are uncertain (you could call that a review).

@BenLinders

You mentioned that "committing to the main branch as chiseling something into stone" which "cannot be undone. What about doing a roll back? "Shouldn't be to hard to do with modern version management tools.

Thomas, I get your point on the roll back. If you wait too long, things can become tricky to correct, but not impossible.

I found a thread on stackoverflow which also explores undoing with Git: stackoverflow.com/questions/927358/how-do-you-u...

But, if a incomplete or faulty commit is done on a feature branch I guess you have similar problems. There might be less people involved though, only the ones working on the feature. On the other hand, given that there are less people working on that branch, it may also take longer to discover it. Preventing such problems would be best, I guess we agree on that.