This project has moved. For the latest updates, please go here.

Branching Strategy for Bug fixes?

Mar 21, 2011 at 7:49 PM

Right now my company has a major release every 3 months.  In between major releases we have monthly maintenance releases. (Similar to the "Advanced Branching Plan") Our monthly release usually is limited to high priority bugs based on customer complaints and business needs.

Scenario:
Lets say there is a production bug reported by a customer shortly after a major release.  It hasn't been determined if this bug will be included in a hotfix, the next maintenance release, or the next major release. What branch should the developer make their changes?

We currently work bugs in our development branch if we don't know if the fix will be part of the next maintenance release. The problem is that this often leads to cherry picking changesets out of fixed bugs and new development work. One idea we were throwing around was to create a new branch called "Bugs" from the parent "Main". This would separate our sprint work from customer defects.  The thought was that this would help separate sprint work from production bug fixes.  I am just curious if this is a bad idea or if anyone has any other ideas? 

 

            Main
   --------------------------------
   |                      |                |
Development   Bugs       Maint 2
                                          |
                                    Maint 1
                                          |
                                    Release

 

 

Mar 21, 2011 at 9:06 PM
Edited Mar 21, 2011 at 11:24 PM

Key excerpt from above:  "It hasn't been determined if this bug will be included in a hotfix, the next maintenance release, or the next major release."

It would really really help to triage the bug thoroughly and have project owner and/or Dev+Test+PM commit to a release timeframe before coding begins. Triage process and management commitment can reduce if not eliminate this uncertainty. If triage commitment is not possible (release timeframe is unknown) then I would assume and prepare for the worst case: The bug might require an emergency hotfix release.

Emergency hotfix branch: Create a dedicated hotfix##### branch as a child of Release for each bug that might need to be released separately.

Develop and test the fix directly off Release safely isolates changes from any other code that hasn't been released.  OK to develop 2+ bugs in same hotfix if related, but they cannot ship separately and cannot be selectively rolled back without introducing cherry picking risks. If you never have concurrent hotfixes under development you could use a single "Hotfix" branch off Release... but chances are as soon as you set this up you'll have two concurrent high-priority bugs without knowing which one will need to ship first. Another risk is one hotfix getting stuck in development or testing which delays the release of all other hotfixes in the same branch.

Branch name "hotfix#####" should be unique. For example, ##### could be the Work Item number associated with tracking this bug or specific hotfix. Or some clever "Hotfix<Feature><Major>.<Minor>.<hotfix#Increment>"

  1. If the fix truly is a hotfix then your branching is proper for a clean release:
    1. FI (Forward Integrate) from Release to hotfix##### (just to make sure any other hotfixes are included)
    2. Final test (required if FI from step 1 brought in any changes)
    3. RI (Reverse Integrate) from Hotfix##### to the Release branch

      Once fix is accepted in Release then merge it back through the tree so all future releases contain the fix (or intentionally omit it by manual merge resolution):
    4. Lock/close the hotfix##### branch.
    5. RI-only from Release to Maint1, then RI-only from Maint1 to Maint2, then RI from Maint2 to Main.
      IMPORTANT: Do not FI from Maint1, Maint2, or Main (before or after RI) because that would merge pre-relesed code to the release branch!
    6. FI from Main to Development branch
  2. If the bug fix is decided to be part of Maint1 (,Maint2,) or Development then do a baseless merge to that branch and then lock/close the hotfix##### branch.
    Be carefult to manually review any merge conflicts. Don't worry about

Keep in mind that a hotfix candidate should be very tactical which makes baseless merge less risky for scenario #2. If the bug fix is more complex in scope then immediately push back hard on even considering the change as a hotfix release. Sometimes it is better to live with the devil you know. I was once told that the odds are good that you will introduce one new bug for every two bug fixes. The big question is whether that one regression is worse than what you tried to fix :-).

Multiple hotfixes can be developed concurrently. Look at the "Branch by Feature" ("Multi-Feature Teams scenario" in TFSBranchingGuideIII) for basic diagram... only think of Release as the main branch for these "hotfix" features.

RE: Adding a Bugs branch as a child of Main branch

I don't think adding a Bugs branch as a child of Main will give you a clean route to Release. When you branch Bugs from Main will it have only the same code that Release branch has... or potentially some other stuff from main?

Caution: If you merge a bugfix through Main you can pick up other changes already in Main. To avoid this you'd have to cherry pick merge just the bugfix files. If any of those files are already changed in Main then you'd have to roll back changes in Main, then merge through (very careful not to pick up unintended changes), then restore the rolled back change(s) to Main. To see what that looks like you can look up the "Branch by Quality" pattern (which is NOT in the TFS Branching guide, possibly By Design). While this is all possible, I am in the process of moving our branches away from "Branch by Quality" to avoid risk and work required to temporarily roll back changes in a branch just to ship a fix.

Even if you get through Main without picking up unexpected changes you'll need to repeat the same careful process to get through Maint2 and Maint1. At this point your best option would probably be to do a baseless merge directly into Release, then RI change back through to Main etc... which would be better done by starting with a short-lived hotfix branch directly off Release.

I hope this is helpful (or at least encourages more useful comments from others). Good luck!

- Zephan

Developer
Mar 21, 2011 at 11:06 PM

In the Rangers Branching Guidance, the Release branches have a specific relationship to each other. In each of the plans, the Release branch is locked down (made read-only) to capture each major and / or minor release shipped to a customer. Since the Release branch is read-only it must be at the lowest level in the release branch hierachy (with Main at the top). If you are doing post-release servicing, then we suggest having a Servicing branch between Main and Release or even a Service Pack branch (as child of Main) and a Hotfix branch (as child of ServicePack) with Release as child of Hotfix (and then made read-only.

Having a hotfix branch as a child of release will make it impossible to move a hotfix to Main in order to have it be part of the next minor or major release. If you review the advanced branch plan, therefore you will see a branch for hotfixes (where all post-release bugs are fixed in the current release). These hot fixes can be Reverse Integrated from the hotfix branch to the service pack branch, in the event you want to roll-up one or more hotfixes into a more thoroughly-tested Service Pack. From there the hot fixes can be Reverse Integrated from the Service Pack branch to Main (which is opened for vNext development when the vCurrent Release branch tree is created.

Your Bugs branch (as a child of Main corresponds to the HotFix or Servicing branch in the advanced or standard branch plan, respectively. The primary difference is that the Release branch is a child of the hotfix or servicing branch, rather than in a separate branching structure. This allows you to have mutliple active release sets (consisting of Service Pack, Hotfix, Release branches) for each of the minor or major releases you need to support in parallel. The hotfix would be applied to the specific release the bug is found in, and then merged (RI) to Main and possibly into vNext development branches.

This structure does, in fact, separate your sprint work (in a sprint development branch) from the Release and Post release bug fixing. It also allows Main to remain as stable as possible. It also means you won't need to cherry pick changes (avoid cherry picking if at all possible).

Since the development branches are working on vNext work while vCurrent is released, I discourage you from fixing bugs in vCurrent in the vNext development branch. If you released sprint 1, fix defects in sprint 1 on the release side, and fix bugs in sprint 2 in the development side (vNext).

I am not sure what the purpose of maint1 and maint2 branches are (between your Main and Release).

My difference of opinion with Zephan's suggestions are that I never make a hotfix branch a child of release (for reasons stated above). To the contrary Release is a child of hotfix. At the time you create the release, the contents of hotfix and relase are the same. Release is made read-only and Hotfix is available for making defect fixes against what was released.

The problem with inverting the structure, is you cannot move a hotfix to Main without going through Release, and doing so means you no longer have a copy of the code as released. If you don't want a locked-down read-only release branch, then skip the hotfix branch altogether, make the hotfixes in the release branch directly (obviously you would not be able to make release read-only in this scenario).

Regards,
Bill Heys
VS ALM Ranger

 

 

Developer
Mar 21, 2011 at 11:12 PM

Zephan,

    "To see what that looks like you can look up the "Branch by Quality" pattern (which is NOT in the TFS Branching guide, possibly By Design)." - Exactly

Regards,
Bill Heys

 

Mar 22, 2011 at 5:34 PM

I am seeing the advantage of having production bug fixes in the Hotfix branch of the Release Vehicle, however I am still a little confused on how to handle multiple service packs.

So for example lets say we had a major release in February with SP1 planned for March and SP2 planned for April. Release branch is locked down and (RI) back to Main.  Post February release there are 8 production bugs reported.  Developers fix bugs in Hotfix branch.

Major Release - February
SP1  - March
SP2  - April
HotFix - contains fixes for bugs 1,2,3,4,5,6,7,8

After bugs have been completed in the Hotfix branch Management determines that:
         bugs 1, 2, 3 need to be in SP1 (March)
         bugs 4, 5, 7 need to be in SP2 (April)
         bugs 7, 8 will go with the next version

Is there a branching strategy that can encompass all this in the Release Vehicle?

 

 

Developer
Mar 22, 2011 at 6:49 PM
Edited Mar 22, 2011 at 8:39 PM

Let's use the following scenario:

In February, you release v1.0, that has been under development. While it was being developed, v1.0 was vNext. Once it is released, it now becomes vCurrent, and development begins on v2.0 (the new vNext)

So when you release vNext, you create (all at the same time) Service Pack branch for Release 1.0 (child of Main), Hotfix branch for Release 1.0 (child of Service Pack) and Release branch for Release 1.0 (made read-only)

Presumably you have some post-release fixes to production for vCurrent. You fix bugs 1,2, and three in the Hotfix branch. You decide to roll-up these fixes into a Service Pack.. so you Merge (Reverse Integrate) bugs 1,2, and 3 into the Service Pack branch for Release 1.0. After thorough testing you release this Service Pack to customers.

AT THIS POINT (March), you could choose to make a new branch for the Released Service Pack (for example, branch Service Pack 1.0 to Hotfix 1.0.1 and Release 1.0.1 and make Release 1.0.1 read-only.

Now you find more bugs to fix, so you fix these bugs in the hotfix branch (for wither Release 1.0 or Release 1.0.1 depending on whether the bug is in the original release (1.0), or the service pack (release 1.0.1)

Note that you do not need a new branch for the service pack ... it continues, once Service pack 1 is shipped in March to be available for stabilizing Service pack 2.

bugs can be fixed in either the Hotfix branch for Release 1.0 or the Hotfix branch for Release 1.0.1 (SP1).

Now (April) you decide to ship SP2 (so you merge any changes in eithe hotfix branch below the Service Pack branch for Release 1.0.x up to the Service Pack branch (RI). You stabilize the bugs (4,5,6) and ship Service Pack 2.

Same process as in March, you make a new HotFix branch for this new Service Pack, and a new Release branch for Release 1.0.2 (SP2). Make the Release branch Read only.

At this point you have ONE Service Pack branch and three Hotfix branches (1.0, 1.0.1, and 1.0.2), and three Release branches (1.0, 1.0.1, and 1.0.2).

Bugs 7 and 8 get fixed in one of these three hotfix branches. They get merges (RI) to the Service Pack Branch and either stabilized for another SP (SP3) or merged again (RI) to Main where they are part of the next Major/Minor release.

From Main they get merged (FI) to the vNext development branches.

I hope this helps.

I just posted this thread on my blog: http://blogs.msdn.com/b/billheys/archive/2011/03/22/where-do-i-fix-a-production-defect.aspx 

Regards,
Bill Heys
VS ALM Ranger.

Apr 15, 2011 at 3:07 PM

Thanks for the previous posts. They have been extremely helpful.  

I was thinking of taking a slightly different approach for the release vehicle and was wondering what your thoughts are.

Using the senerio discussed previously, my team will probably never completely merge all bugs fixed in a Hotfix branch(e.g. 1.0) into the next Service Pack release (e.g. 1.0.1). There will always have to be some cherry picking done.  Not ideal but it is what it is.

Instead of having to maintain multiple(3) hotfix branches I was thinking of creating one Fixes branch that would be a child of Service Pack 1.0.  When a release is complete (e.g 1.0) would be merged (RI) to Service Pack 1.0 and Main.  Service Pack 1.0 would then merge (FI) to Fixes branch.  

When it is time to branch Release 1.0.1 any fixes identified to be in the release would be merged (RI) from Fixes to Service Pack 1.0 before creating the new branch.

____________________Service Pack 1.0_____________________
|                     |                                    |              |     
Release 1.0     Release 1.0.1     Release 1.0.2     Fixes

Developer
Apr 15, 2011 at 3:26 PM

If you make the three Release branches read-only, there is no place to make a hotfix for Release 1.0 separate from Release 1.0.1 and / or Release 1.0.2. Having separate Hotfix branches between the Service Pack branch and the Release branches gives you a place to make and isolate hot fixes for each release.

Regards,
Bill Heys
VS ALM Ranger

Apr 15, 2011 at 5:23 PM

If a hotfix were to come up after Release 1.0 and before Release 1.0.1 a new branch would be created from Release 1.0 (e.g. Release 1.0 HotFix).  Changes would be made in the Release 1.0 HotFix branch.  After hotfix is released Release 1.0 Hotfix would be merged (RI) to Release 1.0 and Service Pack 1.0.

Release 1.0 would always consist of the latest code in production until Release 1.0.1. 

____________________Service Pack 1.0_____________________
|                     |                                    |              |     
Release 1.0     Release 1.0.1     Release 1.0.2     Fixes

       |

Release 1.0 Hotfix

The Fixes branch would consist of production bugs that have been fixed, but it is unknown if these bugs will be included in Release 1.0.1 or Release 1.0.2 or the next major release.  Service Pack 1.0 would be allowed to merge (FI) the latest code to the Fixes branch as needed.   This would allow the Fixes to have the previous release's code making easier merging (RI) to Service Pack 1.0 bugs that are required for the next release.  This also would eliminate having to make merges (RI) from Hotfix branches that are not current with Service Pack 1.0.  (e.g.  Release 1.0 and 1.0.1 are done.  Fixed bugs from Hotfix 1.0 need to be merged to Service Pack 1.0 to be included in Release 1.0.2)