Releasing untested features using git flow

Question

not quite sure how to put that into the title, because the issue is a bit specific.

Basically, we're using gitflow (slight variation of it) in our application. So, this means we have the following branches

• master (PROD)
• uat (STAGING)
• dev (TEST/QA)

We create all our features from dev. The developer creates a PR for their feature branch feature/1234 against dev. Once it's merged into dev, the changes are deployed to TEST/QA

At the end of every sprint, we deploy from dev to uat to put the changes onto the staging server. The staging server is actively tested by many people, that's why we have a specific day in the sprint for deploying where it's expected that maintenance happens.

Now the issue is the following. Let's say, 1 day before the sprint ends and we do the deployment, a feature has been merged onto dev, because code review and local teseting were fine.

However, QA found a small issue or maybe didn't have the time to test it yet.

When we now deploy to uat by creating release-branch off of dev we would also push the changes that aren't QA Done yet onto UAT. This is of course an issue.

I'm wondering how we can cancel out this issue? I have multiple thoughts

• Put a lock on dev one day before the sprint, so that nobody can merge anything to dev anymore when we see all tickets are QA done. However, we would lose one day of development with that, because if on that day we would finish sth. if wouldn't get deployed
• Revert the merge commits of those that should not be deployed, but this feels like a workaround
• Create release branch off uat and cherry pick the merge commits, but tbh, I don't like using cherry-pick for such a case as it just applies the diff again instead of the commit itself
• Only deploy to dev when QA has approved the task before. That way, there is only one feature on dev and we will only merge another if it's QA done, however, this works only if we have 1 qa employee. If there's 2, then what would the other test? Also, we would get a pile of open pull requests that are basically ready, but not merged and then we always have to keep that in mind and it could still happen that the last feature that was merged will not get QA done state until release starts
• Last option would be a release based on feature branch. And only if the feature branch release is successful we merge it into dev and then we can really create pr from dev -> uat and only the changes that have to be in there are deployed. However, we have a microservice architecture and are 5-6 teams. If we would create extra infrastructure for this case it would be super expensive if we want to deploy muiltiple at the same time.

So I don't know the best way. I just knoow that we have issues every release day that we are not sure if we can deploy, if QA will get it done, if QA will have findings and the deployment day is always super stressful

• Stack Exchange Broke The Law: Why don't you just deploy from an older version of dev that has been tested?
• Flater: @user253751 You're assuming things get finished testing in the same order that they are committed. That is not a given.

Answer 1

It feels like one main issue, with one minor issue causing complications.

1. Do not merge anything into dev unless it can be properly tested before the end of the sprint.

   If it cannot be fully tested, hold back the merge. The decision to hold something back requires consultation with QA.

2. In the case you absolutely cannot avoid merging untested code into dev, create the release branch off of a last-known good commit on dev. You don't always need to blindly do git branch releaseX dev. Specify a commit Id or tag representing that last good commit on dev before the untested code was checked in: git branch releaseX asdf7683tg7

When fixing something after the sprint, fix it based on the release branch. Then merge the release branch into dev and any other relevant release branches. Don't cherry-pick commits, because that creates new commit objects with no shared history. Merge conflicts will need to be resolved multiple times, which could lead to inconsistent conflict resolutions.

• Alex Sh.: How can I give a version for QA to test without merging tasks into one branch (dev)? We can't test features separately because they use the same codebase, which could be change during working on one of the features, isnt't?
• Greg Burghardt: @AlexSh. You can merge into dev, but only if it can be tested before the end of the sprint. Otherwise hold off on merging that code in. This implies developers and QA testers are coordinating efforts.

Answer 2

When an issue is found in any environment, I would recommend creating your "feature" branch off of that environment's branch. This is effectively the hotfix branch. You are not necessarily introducing new features, but providing patches to ensure that intended functionality works as expected or to remove a feature if it cannot be made to work as expected within your release schedule.

This will ensure that teams working on future work aren't blocked from integrating their work and won't clutter up the history like reverts (and perhaps reverts-of-reverts).

It's hard to say if the phase-gates that you have are appropriate. Assuming they are, you may want to think about how to make them harder. That is, if you're going to have the gate of completing QA before being promoted to staging, don't promote something that hasn't been through the QA process. Of course, there are probably better ways of working than a hard phase-gate, but that would be some significant shifts in the way of working that would take an investment to implement.

Answer 3

Given how your deployment process works, you likely will not be able to adequately solve this scenario until your team is able to deploy much more frequently. There will always be last-minute issues found - sometimes even after the change has ostensibly passed UAT.

What happens if QA finds an issue with one of the changes the day before deployment? Are you going to deploy all the other changes? How do you know that the version of your app that has all but the one change actually works since no one has tested it?

The systemic problem of not being able to deploy as soon as a change is marked ready for production is what should be addressed. The fewer changes you have waiting to be deployed, the less pain you will have.

Answer 4

So the uat branch is deployed to the STAGING environment, and that's where bugs are found? Why not create a bugfix/* branch off the uat branch instead of creating a release branch off of dev? When resolved, you merge to both uat and dev, then redeploy uat to STAGING.