TFS Branch questions

Nov 18, 2010 at 4:39 AM

I have the challenge of designing a new approach to our code/branch management in TFS 2005.    Our current approach has static code branches that represent various test environments.    We built this approach before the rangers and with little documentation. It lacks the flexibility we need to support multiple clients and concurrent development and release testing.

I have studied the Standard and advance branch concepts… and had some questions..

1.        Where do I build from for test (release 1, release 2, etc). I think you build from Main, I may have two or three test code bases at the same time.   Release 1 current with 2 and maybe 3.  

2.       Do you deploy production from a different release branch each time?  What tool/approach do you use to make sure that you build production from the correct branch? We currently build from the same branch (prod) all the time. If you do a hot fix for release 1 do you make the changes in the “release 1 fix branch” and merge to the release branch? How do you manage security for these branches? Do you unlock the hot fix branch only for the developer working on the fix?

3.        I may need to support two test environments UAT1 and UAT2 – based on client needs.   What is the best approach to automating this build process when my code source may come from different branches?

4.       What triggers the move from Main (UAT) to the release branch? Is this after UAT is complete or before? What triggers code to merge from Devl to Main?

6. Merges can be complex. I am very concerned about the time required for merging code and resolving issues. What are the tips and tricks to making this work?   Specifically I am concerned about merging the DB Schema.  DBPRO is good, but it doesn’t do the schema to our standards, we do the DB schema merges manually (compare, asses and script).

Any help you can provide would be appreciated.

Developer
Nov 18, 2010 at 5:27 PM

If you start by looking at the basic branch plan, here we have three branches: Main, Development, and Release. Let's suggest that there are three releases being supported (Release 1.0 and Release 2.0 are both in production, while Release 3.0 is in development) Let's call Release 2.0 the *current* release (vCurrent). This makes Release 1.0 vCurrent -1 and next release is Release 3.0 (vNext), which is not yet in production.

The only difference between the basic branch plan and the standard branch plan is that we have added a new *level* of branching on the Release side, the Service Pack branch. This is the branch where post-release hot fixes and / or service pack changes are made.

Let's suggest that you are getting to the point where you are finalizing vNext (Release 3.0). All of the enhancements for this release have been tested in the Development branch and, when they are ready to be released, are merged into the Main branch. What triggers a merge from Development to Main? The code in Development must be *feature complete* and tested. The code in Development must pass pre-established quality gates before it can be considered *Ready to be stabilized for Release* or *Ready to Share* or *Ready for Integration*. You need to determine what you establish for quality gates (typically you track code quality indicators such as rates of new bugs reported, rates of bugs fixed, rates of bugs re-tested and marked complete, etc.). What this means is you should have a formal QA process to test code in the Development branch *before* it is merged to Main. Bugs in development should be tracked and reported. When you get to the point where the rate of new bugs reported declines (toward zero) and the number of outstanding bugs declines (towards zero). When you reach a predetermined level of quality, then the code is ready to be released. You should label this code (milestone labels are useful). Then you should merge the code from Development to Main.

The same process applies to code in the Main branch. After the new features are merged for vNext stabilization in Main, you continue to stabilize the code in Main until you reach a pre-determined level of qualtity. When you reach this level of quality, the code is ready to be shipped. You would not create a Release branch until the code is ready to be shipped and released. The assumption here is that you would complete all testing including UAT before you branch from Main to Release (or Main to SP and SP to Release).

I look at supporting test environments as a deployment issue, not so much a build issue (although the two are related). Rather than have separate branches for UAT1 and UAT2 (as some suggest), I deploy code from Main to each of the UAT environments. This allows you to control which build is in each UAT environment and allows bugs to be associated with specific builds. It is difficult for QA if the code in UAT is continually being changed and built. Deployments to QA (UAT) should be controlled so they are testing a relatively stable code base.

Each branch should have it's own automated build process. There are many flavors of build that you can consider. However, I would strongly suggest that you consider moving to TFS 2005 to TFS 2010 (or even TFS 2008) becuase of improvements that have been made to MSBuild and Team Build since 2005. Where supported, I recommend implementing Continuous Integration (CI) builds. This means that whenever a developer check-in code to a Development branch, an automated build is automatically triggered. There is a flavor of this in TFS 2010 - the rolling builds- that queues up check-in triggered builds so that a new build waits until a prior build completes. There is even an option for delaying the next build for a specified period of time so you do not overload the build server during times when there a many check-in in a short period of time.

We recommend doing daily builds in the Main branch. After a good daily build in Main, we recommend merging Main to the Development branch(es). This reduces the degree to which Main and Development become out of sync. When you do this on a daily basis, there will be more frequent merges with fewer merge conflicts each time. If you wait a long period of time between merges from Main to Development you will have the *big bang* effect where a lot of merge conflicts may need to be resolved at the same time.

In short, you would deploy from Development to QA for feature testing (you may have more than one feature branch in development). You would deploy from Main to QA for integration testing and for stabilizing the code for release.

How do you ensure you build production from the correct branch? This is why you have a build master. You do not want every developer having the ability to do builds on the Main branch. The build master should be knowledgeable and experienced enough to follow proper procedures when building for production release. If you have the advanced branch plan (with three levels of branches on the release side), you should be aware that these three branches are created at the same point in time. When the code is ready to ship, branch Main to Service Pack. Branch Service Pack to Hot Fix, and Branch Hot Fix to Release read-only. You should not have bidirectional branching on the release side of the picture (e.g. you would not merge a hot fix down into the release branch, but you could merge a hot fix up into the service pack branch and then up to main to bring it forward).

On typical projects you have a variety of roles. On larger teams you might have developers grouped into teams with technical team leads. Then you have focused roles such as project manager, technical team lead, branching lead, build master, release master, etc. It is difficult to make specific security recommendations that apply generally to a wide variety of situations. At a minimum developers probably need check-in and build privleges on a development branch. Nobody should have check-in privileges on a Main branch. The build master should have build permissions on the Main branch, etc.

As for locking / unlocking the hot fix branch - this is probably not necessary. Controlling merges involving the hot fix branch seems to me to make more sense.

I am not an expert on DBPRO. I would be interested in hearing of your experiences in this area. What are the shortcomings or challenges you face when managing the schema with DBPro. Perhaps we can forward some ideas along to the product group.

Regards,
Bill Heys
VS ALM Ranger

 

Nov 22, 2010 at 3:43 PM

Bill,  Great Stuff - thanks for the quick response...

A few more questions…

1. Ref: “Rather than have separate branches for UAT1 and UAT2 (as some suggest), I deploy code from Main to each of the UAT environments. This allows you to control which build is in each UAT environment and allows bugs to be associated with specific builds. It is difficult for QA if the code in UAT is continually being changed and built.”

1. Q:  I assume that Main is always VNext.  The example of UAT1, UAT2, etc.  I assume this is the same Vnext code base? - in different test environments?.   Vnext +1, +2 would be in two development branches and merged to Main when Main is ready for Vnext +1 (now Vnext).  Is it best practice to merge from MAIN to development and then merge development to MAIN when ready for the next vNext.

2.Q: Do you recommend creating development branches by release or by feature? We are often challenged with projects that are targeted for a release but don’t make it for numerous reasons. I also want to test feature dependent releases concurrently.   I have the impression that a development branch should be Vnext+1, etc.

3. ref: “Nobody should have check-in privileges on a Main branch”

3. Q: How do you make bug fixes in Main? Do you create a Main bug fix branch or use a development branch?

4. Regarding DBPRO, I can get that list to you. For the sake of discussion, let’s say that DBPRO does not exist. What is the best practice for merging DB Schema between the various code branches?

Thanks!

Tod

Developer
Nov 22, 2010 at 5:06 PM

If you are working on vNext, vNext +1, vNext +2 all at the same time, you would most likely NEED to have separate Development branches for each of these. It is unusual for companies to be dealing with so many future versions at one time. Perhaps you have frequent releases that require longer development time frames.

Main is typically dedicated to stabilizing the next release (vNext) before it is released to production. In theory, you would do daily builds in Main and, following each good daily build, do merges (FI) with the development branch(es). You would not merge the vNext development branch with Main until the vNext development branch is ready to be stabilized for release. At this point merge vNext Development to Main (Main is now used for stabilizing vNext). There may be a period of time between when you release a version (vCurrent) from Main and you are ready to stabilize vNext. During this period of time Main will be vCurrent (with fixes).

With respect to bug fixes. I prefer the Standard or Advanced branch plan. If you are making bug fixes to released code, make the changes to the SP or Hotfix branch associated with that release, and merge (ri) this hotfix up to Main. If the bug fix is found during stabilization of code in Main, make the bug fix in the development branch and merge (RI) the fix down to Main. If you are feature testing code in a development branch (before it is ready to merge to Main), make the fix directly in that development branch.

I am not a DBPro exepert, but I will try to get some anwers for you.

Regards,
Bill Heys
VS ALM Ranger

Nov 22, 2010 at 9:56 PM

Great stuff,

We do monthly releases with many projects that take more than a month. We often need to start v+2 or even V+3 concurrent with V.   We currently manage this by having one development branch and promoting to “main” by project/work item instead of by branch. It certainly causes a few problems, but it does work.

I assume that when you merge, you are referring to an entire branch merge? Development branch xx merged to MAIN would be the entire branch. Do you reuse the development branch xx for the next feature or leave it as a dead branch? You can assume that we will need 2 or 3 development branches for future releases.

Let’s say that you have 6 features for Vnext +1 currently in development branch 01.  Feature 5 gets delayed.   Do you roll feature 5 changes out of development branch 01 and merge with development branch 02 (Vnext +2). Then merge the entire Development branch 01 to Main?

One more question on supporting the production code.   In the Advanced Branch plan, I have my hot fix branch where I make any urgent changes.   I am not quite clear on how I build these changes to prod.  Do you build fixes from the hot fix branch and overlay those files in the production environment? Some months could require 2 or 3 hot fixes and associated deployments to production. What code do I pull to recreate the current production code (release + service paks(s) + hot fixes).

Thanks much,

Tod

Developer
Nov 23, 2010 at 9:26 PM

Tod,

It seems to me you would be better off having a separate development branch for each version you are working on. This offers several advantages over the alternative of promoting (merging) to main by project/work item.

  • One problem with cherry picking changes by work item is that is too easy to pick up partial changes. For example, if a developer makes a change in v+1 to a particular class (file). Then later makes a change to fix a bug, and finally makes another change to fix a bug. You need to be sure you merge all of these related changesets to Main or you will potentially be merging buggy code to Main.
  • What happens if the developer forgets to associate one of these changesets to the correct project / work item? This changeset might not get merged.
  • If all of your developers are working in the same branch but are working on different releases, how do you avoid conflicting changes. For example, suppose you have an interface change that must be implemented for v+2. If that change is made in the same branch with code for V+1, you will introduce a nightmare situation. This is a major reason for feature team isolation in development (or for branching development for release). As an aside, you may hear some people talk about branching for release without being clear about whether they are branching for release development or for release to production. In this case I am referring to having separate development branches per release (v+1 and v+2) as well as separate release branches when the time comes to ship.
  • With cherry picking you run the risk of picking too many or too few changesets to merge. This introduces instability in the target branch. You also have situations where a cherry picked change gets merged again on top of code that has moved past that point.

By having separate branches for release and perhaps for features within a release you are better able to accomodate changes to the schedule (such as where feature 5 is not ready when the other features for a release are). It seems it might be quite challenging to identify all of the changesets associated with feature 5 and excluding them from a merge.

With respect to hot fixes.. this is one of the more challenging questions to answer. When a company is shipping a product ot multiple customers the situation becomes more complicated than when you only have one version in production (such as where you are the customer of your software).

Let's suggest that you ship V1.0 to ten customers. The release branch for v1.0 (read-only) is a snapshot of what was initially shipped to those ten customers. Typically, however, when you do hot fixes you are shipping a patch to fix a specific problem for a specific customer. So you create a hotfix for one customer. Later you create a different hotfix for a different customer. You now have three different *versions* of your code running. You might consider creating a new hotfix branch for each new hot fix or, in the case of where you are your only customer, you could consider reusing the hotfix branch. In this scenario the hotfix branch contains all of the hot fixes and would be the latest version of your production code.

It is challenging to answer the question of *how do I build a hotfix*. Typically hot fixes are deployed as patches (MSUs or individual DLLs that have changed) rather than full installs (MSIs). Given all the variables at play I am not sure how to best advise you here.

Regards,
Bill Heys
VS ALM Ranger