Release strategy for multiple Git feature branches being tested simultaneously

Question

We have three environments: Dev, UAT and Prod. We use TFS to schedule releases of our master branch into Dev for our internal verification, then UAT for business verification, and of course finally to Prod once approved.

We've recently adopted a new lightweight Git branching strategy as follows:

master is always prod-ready. At any point master should be able to be deployed to production.

All new development is done in a separate feature (topic) branch as follows:

1. Create a new feature branch off master, call it FeatureA
2. Develop FeatureA until completion
3. Once FeatureA is finished, release it to Dev, and then UAT
4. Once the business signs off on FeatureA in UAT, it's considered prod-ready. Merge FeatureA into master, then deploy the new master branch to Dev then UAT. During the way, "smoke test" the branch in UAT to ensure the resulting merge into master didn't cause any unforeseen side-effects. Once smoke-tested, release to Prod.

The problem we're coming across right now is that we may have multiple features being developed in parallel, all of which could potentially need to be deployed to the test environment for verification at the same time. The approach we've taken to solving this problem is:

If FeatureA and FeatureB need to be in UAT at the same time, then:

1. Create a new branch, FeatureAandB, which will encompass both features
2. Merge FeatureA into FeatureAandB
3. Merge FeatureB into FeatureAandB
4. Release FeatureAandB to Dev, then UAT

The downside to this is that it's unlikely both FeatureA and FeatureB will be UAT verified at the same time. If FeatureA is verified and FeatureB is not, we need to release FeatureA to prod without FeatureB. What we've discussed in this scenario is to:

1. Merge FeatureA (not the joint branch, but just FeatureA) into master
2. Release master to Dev, then UAT for a quick smoke-test, and finally Prod
3. Once in prod, re-release just FeatureB to Dev then UAT so testing can continue.

The downside to this is that it directly impacts any testing for FeatureB, and potentially unwinds any work the testers have accomplished with FeatureB.

How do you manage multiple features living simultaneously in each environment and being released potentially independent of one another? We can mitigate the issue a little more if we have multiple environments, or turn-around UAT testing much quicker, but at the end of the day the same problem can exist.

I'm not opposed to hearing alternative branching strategies, either.

• mmathis: How often does this situation come up, and how long does it take Dev and UAT to test?
• StoriKnow: @mmathis historically, not very often, but we've recently dedicated more resources to this app and have several features being developed in parallel, so I anticipate this happening more frequently than in the past. As for length of time in UAT, that varies by the complexity of the feature and how involved testing must be, as well as the availability of the stakeholders requesting said feature (stakeholder of FeatureA isn't guaranteed to be the same as FeatureB, either).

Answer 1

Ahh. Congratulations! You’ve destroyed one bottleneck and discovered the next one! Now it’s time to look at actually continuously integrating your code. As you’ve found out, it’s hard to continuously deliver when your code under dev isn’t being continuously integrated, but how do you make this work?

No more feature branches.

No. Seriously. No more feature branches.

It’s time to introduce feature toggles into your system. Instead of delivering new features by delivering code, you need to have complete control over when a feature is turned on for a given environment. In other words, you want to have environment specific configuration that turns features on and off. This decouples code releases from feature releases.

• Dan Cornilescu: Potentially of interest would also be branch by abstraction
• StoriKnow: Oddly enough, this is what we had in place, and what we were trying to get away from with the main reason being "if the code isn't prod ready, it shouldn't go to prod, even behind a feature flag". This originally came from the results of an audit. However, the points you make are absolutely correct, we're attempting continuous delivery without continuous integration, which just doesn't work. Do you by chance have any resources on hand that could help educate myself around this process, and why it's actually the way to go, or how this doesn't violate any compliance concerns?
• RubberDuck: I have no idea what kind of regulations you have to deal with, so I really don’t know. Sorry.
• RubberDuck: What kind of audit?
• StoriKnow: Of course: SOX compliance. However, I may have spoken too soon when I brought that to your attention, as I believe we already satisfied their concerns by implementing an approval process from a non-dev before deployment to each of our environments. With that said, I think this is honestly the best approach, it really becomes a challenge of convincing the team and management. Thanks for your insight.

Answer 2

Test features separately and in order, that's really the best way to do it. Usually whichever one is done first gets tested first, but business needs could change that priority. When FeatureA is accepted, it's merged into the main branch (dev, UAT, or master in your case, as applicable), that branch is merged into FeatureB and then feature B is tested. A is tested first, then B, then C, etc, with each feature branch getting updated as other features are tested and accepted.

By testing features separately, you keep the focus on one feature and one team is responsible for fixing any problems. There's a clear responsibility for keeping the code in a good state when handling merge conflicts (in this case, the featureB team needs to merge dev in and deal with any merge conflicts, ensuring that the code from featureA is maintained). Separate features also makes it easy to assign "blame" when a problem is discovered - not the "why did you break this" kind of blame, but the "this feature is causing a problem, remove it from dev until we can sort it out" kind of blame

If you're finding that a lot of FeatureX branches are spent waiting for testing or multiple branches are being completed at the same time, you can either break them up into smaller features, or allocate more resources for testing. Both approaches should decrease the time to test an individual branch.

Similar features that will be completed at the same time may best be developed on a single branch. That's kind of what you propose with the FeatureAandB branch, but in this case, they get accepted or rejected as one. If the features are not similar / related, though, test them separately.

Finally, make sure you're making use of any automation available, so teams aren't waiting on another team for anything. Most build servers can be set to build Pull Request branches, producing a FeatureA+dev build which can go to QA for acceptance and integration testing. The developer simply opens the PR (which also serves as a code review, if desired, and also serves to signal that the feature is ready for testing), and once QA signs off on it, completes it. This then merges the branch into dev automatically. Can do the same for UAT and master, creating PRs for merging dev into UAT, and UAT into master.

You may then be able to reduce the number of branches, and stick with dev and master, plus featureX. Get rid of UAT, as that role is essentially served by dev. Actual development is done on featureX and production is in master. Deploy new code whenever master is updated, or deploy whatever is in master whenever you want to release.

• StoriKnow: Thank you for the detailed response. Unfortunately our environment and way of business doesn't necessarily lend itself to sequential feature testing/ rollout. Our application is a monolith with multiple sub-applications it encompasses, each of which has a different set of stakeholders (or even multiple sets). Along with that, the stakeholders aren't always as quick at completing UAT as we'd like, while some others are. Holding up testing of one feature while another is being tested just won't go over well.
• mmathis: If your alternative is to test multiple features at the same time (FeatureAandB), testing takes twice as long. End result is the same, but testing them separately gives you additional benefits.
• StoriKnow: Well, not necessarily. If we deploy FeatureAandB in parallel, and they have two different sets of stakeholders, then they can be tested in parallel as well. The sequential testing you suggest really shines when FeatureA and FeatureB are both going to be tested by the same group.
• Dan Cornilescu: To me this encourages long-lived branches, which increase risk and costs and decrease overall development velocity. Waterfall, in a way. It used to be the norm, but these days we have CI/CD - agile enabler and generally considered a superior development approach.
• Juh_: An underlying issue I can see here is that the testing is "too long", with respect to development. So making it is faster would clearly improve the overall process (which ever way you implement it, independent feature being best though). So, either add man power or automate as much as you can the testing part. If you automate enough testing, you can accept feature A before it is user tested (because nothing serious will be broken), not blocking feature B. Developing a proper test system is a long run, never ending, work. So start soon