This project has moved. For the latest updates, please go here.

Shared roots - stay in the islands or move to the Main land?

Feb 12, 2011 at 10:46 PM

Q1. Should we establish a cross-project (common/shared/root) "Main" branch between our existing TFS Projects that ship to the same Production (enterprise web service extranet) environment?

If cross-project Main branch is a compelling benefit then

  1. Where should cross-project "Main” be located?
  2. How should project-specific INT, QA (Main), and Prod branches be migrated to flow through a new cross-project “Super-main” branch?

 

Q2. What should we do to mitigate risks associated with current disconnected root branches if we don't implement a cross-project Main (or until we get through refactoring existing branches)?

I suspect many teams start out creating a separate TFS Project for each major project. Separate TFS Projects falsely imply needing a separate Main branch for each project, even if projects are related in some way. We release some projects that have components from three separate TFS Projects (each with their own Production branch). It works and we could extend by building "bridge" branches from shared project to consuming projects without having to share a root main, but this pattern isn't described anywhere I've read... which is a bad sign.

What would you recommend given the information below?

Scrum-based SOA Web Dev shop, ~25Devs, shared source increasing, 4 test & 1 production environments, 8+ existing TFS Projects (each with their own root branch), very limited SCM resources, etc. (See MORE INFORMATION for much more detail.)

 

----------------------------------------
MORE INFORMATION

I considered skipping this additional info. If your answer to the above branching question is "it depends on the situation…" then keep reading for additional context.

BRANCHING SITUATION (requirements, constraints, environment etc.)
Branching structure ideally follows development and project needs. Here is the core development situation we have today to help identify preferred branching patterns:

  1. ~25 Developers across 3 countries working on ~4 major projects plus a few minor projects at any given time.
  2. SOA (Service Oriented Architecture) ASP.Net web services (15 and growing), related websites, and some win32 apps.
  3. User Base: up to 10,000 internal customers and partners worldwide connect to Production environment (extranet)
  4. Variable release schedules:
    1. Projects often have different release cycle (not grouped releases like Microsoft Office Suite).
    2. Typically 2 week sprints (each released to INT), but often up to 6 sprints between production releases.
    3. Sustaining work is typically 1 day to 2 weeks per hotfix.
      (This all adds up to needing to release hotfixes outside the normal sprint cycle)
  5. eature-branches under an integration branch for prototype work, multi-sprint features, and multi-dev breaking changes
  6. Shared source code appropriately managed and easily used by several projects
  7. Environments:
    1. Dev (multiple) - multiple dev environments in India, the Netherlands, and USA. Daily deployment (automated) is new model gaining traction but local devbox deployments are still common.
    2. INT - Simplified for testing, full dev access. Deployment done at each sprint exit.
    3. QA - pre-production testing and User Acceptance Testing. RC deployments (~3 month cycle depending on the project)
    4. Production - Release deployments (and occasional low-risk hotfix). ~3 month cycle depending on the project
  8. Release Repository: Archive of each QA & Production release including binaries, deployment instructions with configuration info, specs (project, Design, test,…), test results, …
    1. Audit trail required for what was released to Production and by whom.
  9. No dedicated Software Configuration Manager, Release Manager, or Build Master
    1. SDET3 (already a limited test resource) is currently the acting TFS Administrator and configuration manager (focusing on initial build and deployment automation to enable daily test automation)
    2. This is a significant constraint we have to work with until long-term SCM resource is secured. Hopefully this will be addressed in next several months, but assume "Get a SCM" isn't a viable part of the solution for at least 12 months. We can get tools to mitigate branching and release risks so related suggestions are welcome. (Please disclose if you are related to any third party tool you recommend.)

  

BRANCHING PROCESS FAILURES observed with current configuration:

  1. Release bits built in INT branch + failure to merge to QA branch
    1. FIX = QA require final build is automated (not private build) from release branch (or QA branch if Branch by Quality). Also release from single QA branch rather than from up to 3 separate TFS Project QA branches (in other words move to a common "Main" branch").
  2. Inconsistent merge to Prod branch after final build & release from QA branch.
    1. FIX = QA or Release Manager responsible for copying final (officially tested & signed off) build to release repository.
  3. Dev branches for each developer in each project (branch mania).
    1. FIX = limit branch creation permissions. Also train devs how to easily use workspaces and shelvesets for isolation as well as when to request a temporary feature branch.
  4. Dev "branch" created as folder copy (no branching relationship established = merge risk + no intermediate change history carried to parent branch).
    1. FIX=Clear branching owner + "When & How to branch" guideline.

 

CURRENT BRANCH STRUCTURE (simpilified & generalized)
Separate root "Production" branch for each project - no shared root branch
"Branch by Quality" branching pattern for most projects (Prod->QA->INT[->Dev-Feature])

BRANCH HIERARCHY SUMMARY: Small forest with each TFS Project having one or more tree containing several branches.

Typical tree in each TFS Project: (Prod is "top" root branch)

Prod
    QA
        INT

 Typical Folder structure:

<Project>
    \Dev
        \Dev-Feature1
        …
    \Int
        \Main
    \QA
        \Main
    \Prod
        \Main

Yes, we have 3 branches named "Main" in each TFS Project (over 12 "Main" branches in the TFS collection!). Even more confusing is the TFS Branch Hierarchy diagram showing Main->Main->Main :-). For this post I've ignored the current "Main" branch names as we always refer to "Int\Main" as the "INT" branch (etc.). At a minimum we'll be renaming these to their proper names with next branching refactor.

 

CURRENT TFS PROJECT STRUCTURE: (simplified & generalized - TFS projects at root levels in bold):

Notes: This is basically what we have right now. It is NOT an ideal structure and has several issues to deal with. Much of this structure comes from acquiring projects from other teams or past outsourced vendor sources. We expect to have significantly more shared components and shared library code going forward.

TA = TFS Project "A", TPlatform1 = TFS project "Platform1"

  1. TA - Depends on UI in TSustaining, some web services in TPlatform1
    Prod
        QA
            INT
                Dev-<Feature>
                <DeveloperName> (Moved away from this pattern in 2010)
  2. TB- Depends on BizTalk in TPlatform1, some services in TA
    Prod
        QA
            INT
                Dev-<Feature>   
  3. TC - Depends on some services in TA, some new shared code in TPlatform1, and some UI in TSustaining
    Notes: Newly acquired project. Full branch structure pending common branching guidelines.
    Main
        INT
  4. TPlatform1 - No cross-project source code dependencies (yet)
    Notes: Establishing shared code in same branch
    Prod
        QA
            INT
                Dev-<Feature_Version>
                <DeveloperName> (Moved away from this pattern in 2010)
  5. TRepository - No source dependencies
    Notes: Official project documentation + released binaries stored in here for all releases of all products.
        SharePoint sites are nice for live documents but moves and other events can impact history and audit trail is too soft for key documents.
        Currently use single TFS Project for all release binaries and docs rather than archive stored in each project
        Ops deploys from binaries and deployment instructions checked in to the repository.
        Currently source code used to build release artifacts remains stored only in related TFS Project(s). (We might revisit this to create single convenient package that includes source used to build released binaries, but this isn't a high priority currently.)
  6. TSustaining - No cross-project source code dependencies
       
    Notes: Huge number of old dev branches (30+), some copied folders (not created as branches). Originally created after major insoruce project switched to maintenance mode. No plans to move any new projects to this TFS Project.
    Prod
        QA
            INT
                Dev-<Feature_Version>
  7. TSharedServices - No cross-project source code dependencies
    Notes: It would be great if this was called "Common" so all shared source would make sense to store in here
    Prod
        QA
            INT
                Dev-<Feature>
  8. TTools - Self-contained (Custom deployment tools, etc.)
    No branches defined in this project, but everything is under a single "Main" folder. Rarely changed currently.

  

CURRENT PLAN:
If you've read this far I'm really impressed. I got lost several times just getting myself this far :-). I still need to settle on how to refactor common Main branch and what to do with existing QA and Production branches. I've got draft plans to Implement changes but need to resolve common Main root whether to switch to Branch by Release.

  

RESOURCES:

I've read quite a bit online trying to answer this and related branching questions. Here's a short list of branching resources that have gotten me this far :

  1. Branching for Scrum- Bill Heys (ALM Ranger) 2011.01.18
        Excellent scrum team diagrams and how to scale branching.
        Guidance is "Branch by Feature" with common Main (+ "Branch by Release" for medium and larger teams).
  2. Parallel Feature Teams working on multiple releases in development. Monthly releases to production. - Bill Heys 2011.01.14
       
    Very similar to our branching scenario (3 Web dev teams + 1 Prod env.).
        Guidance is "Branch by Team" to common Main + "Branch by Release" (new release branch for each monthly release of entire Production env).
        Missing discussion about multiple products that ship (not just multiple features of same project). However you can say that a product is a scaled-up "feature" of the release.
  3. Guidance: A Branching Strategy for Scrum Teams - Martin Hinshelwood 2010.04.14
  4. Branching Guidance - Bill Heys and Willy-Peter Schaub (MSDN Magazine) 2011.01
  5. Visual Studio TFS Branching Guide 2010 - ALM Rangers 2010.03.31. Visio template, "The standard branching documentation"
  6. Branching Strategies-2 (Branch by Quality) - Jeff Levinson (Northwest Cadence) 2008
  7. "Agile Branching Web" search in MSDN Forum Team Foundation Server - Version Control

  

REQUEST: Please post your initial thoughts and any partial answers that you have on hand. You are welcome to refine and extend your posts to attempt a "grand unification theory" answer. Thanks in advance for your thoughts!

Developer
Feb 13, 2011 at 8:49 AM

As you note ("If you've read this far I'm really impressed"), this is an extensive and complicated set of questions. I will be taking some time to make sure I understand what you are trying to do, and then I will work with you to make some recommendations.

Regards,
Bill Heys
VS ALM Ranger
http://blogs.msdn.com/b/billheys/ 

Feb 14, 2011 at 7:40 PM

Bill: I really really appreciate your initial reply even though it is "processing...". I was actually hoping you'd chime in. I have found your recent "Branching for Srum" and "Parallel Feature Teams" blog posts to be the most relevant content so far (and extremely useful :-)).

All: Please feel free to post your thoughts on the initial summary Q1 and Q2 without the (admittedly lenghy) context. If you have questions I might have an answer but I don't mind repeating answers to any probing questions if you don't have the time to dig through the MORE INFORMATION section.

Developer
Feb 15, 2011 at 11:02 AM

One of our key recomendations in the Rangers Branching Guidance is to minimize the number of *levels* in the branching hierarchy. Each time you add a level, you double the number of merges (RI and FI) to move a change from the top-level parent to a bottom-level child or vice versa. To get the discussion going, I thought I would give my perspective first on the branching pattern:

Prod
    QA
        INT
            Dev-<Feature>

I view QA as a deployment process, not a branching process. In other words, using your branching structure, I would eliminage the QA branch. Your Prod branch would seem to be similar to the Release branch(es) in our basic branch plan. Your Int branch would seem be similar to the Main branch in our basic branch plan. As you know, Main is where we stabilize code prior to release. I view the stabilization process as one that involves periodic *deployments* from Main to the QA environment. Bugs that are identified by QA are usually fixed in the Dev branch, re-integrated (RI) to Main, and then deployed to QA whenever QA wants or needs a new build. I don't seethe value of having a separate layer in the branch hierarchy for QA.

The second observation I would make would be the relationship to the branches in the hierarchy. For example in the basic branch plan we *start* with the Main branch - where code is stabilized prior to release through periodic or on-demand deployments to QA. From Main, as the top-level parent, we branch to the Development branches (full-children of Main and level 2 of the hierarchy). We merge code from Main to Dev daily (FI), but only merge code from Dev to Main when Dev passes quality gates (ready to release or ready to share with other teams working on the same (vNext) release.

After the code is stabilized in Main and is ready for release, we branch the Release branch(es). These are also full children of Main and at level 2 of the branching hierarchy. The key difference in our model from yours is we can easily support multiple releases. The Release branch is a child of Main. In your branch plan your Prod (Release) branch is at the top of the branching hierarchy. You cannot have more than one parent in a parent-child relationship. So, while you can have multiple Dev branches as children of INT, you cannot have multiple Prod branches as *parent* of QA. Put another way, for any given DEV branch there can be only one INT branch, one QA branch, and one Prod branch. What is the difference between DEV, INT, QA, and PROD? Dev is the least stable branch, where code is developed, checked-in, and tested before being merged with code from other teams and / or released for stabilization testing prior to release. Prod is the most stable branch (in our model it is typically a Release branch and is locked down after release).

QA and INT therefore are intermediate branches that exist for code between the least stable state (DEV) and the most stable state (Prod). I argue you could have several code promotion stages between DEV and PROD. But our recommendation is to have only ONE (the Main branch in our model). If you have multiple feature teams (DEV branches), you could deploy code from any feature branch to QA for feature testing prior to integration with code from other teams.

I will continue to look at your questions and details and will probably make some further comments.

Regards,
Bill Heys
VS ALM Ranger


http://blogs.msdn.com/b/billheys

 

 

 

Feb 24, 2011 at 5:00 AM

Thanks for the guidance Bill. I agree minimizing levels is a best practice. In fact I'll quote your recent response to "Branch by Release vs. Basic Plan" because in some ways it summarizes the entire branching Guide (and because I love how consice it is):

"Start with Main... create as many development branches as you need, when you need them as children of Main. Create as many Release branches as you need when you need them, also as children of Main"

Branch by Quality description specifically mentions it is for supporting one version at a time (though can be adapted by adding branches for each release). However I believe the Basic plan does the same with one less level. Basic plan also reduces naming confusion because QA and Production are environments that cannot feasibly be identical to contents in branches of the same name (unless you replicate your entire environment including OS and HW configs into your branch) :-). I'm fairly convinced after much thought and reading that "Branch by Quality" causes more confusion than it solves. I'm happy to call this detour covered.

OK, back to my original question... which I don't think is described in the Branching Guidelines nor in response above :-D.

Q1. Should we establish a cross-project (common/shared/root) "Main" branch between our existing TFS Projects that ship to the same Production (enterprise web service extranet) environment?

Scenario: Let's say I have 7 (or 20+) TFS projects each with their own set of 3 branches for (Release<-Main->Developer). Should I reparent all the Developer branches to have a single root Main branch?

Answer attempt#1 (extrapolating guidance for individual projects):

Extrapolation: Features can scale to be entire software products. For example, Microsoft Excel can be considered a feature of Office. Office has shared components such as Menu Ribbon, Spell checker, etc. Therefore Excel, Word, Spellcheck, Menu Ribbon, etc. could all be child branches of a single Main. (Excel is big enough it probably needs some feature branches or Team branches off it's primary Development branch.)

Generalization: Any projects that share source code or will likely share source code should have a common Main root branch.
(Projects that share a installer package and release vehicle might also be good candidates to share a common Main branch, but I don't have as much to extrapolate this from.)

Limits*:
Having 7 (or 20+) separate Main branches is fine as long as none of these projects share or will share code with other projects.
Main must have a common quality bar for merges. Any team or product that can't meet this bar must ship from it's own main (at it's own risk without impacting other projects).
...
*This is the part where I need some guidance. Too many pieces shipping through same Main branch could destabilize Main - and block release of other projects.

Process (Draft 1):
1. Create the new root "Main" branch
a) create new root branch in a TFS Project named "Common" or in existing TFS Project where you are consolidating Projects that have or are shared source code. (Consolidating TFS Projects is a full discussion on it's own - and there are good discussions about pros & cons elsewhere.)
b) If you can't rename all the existing "Main" branches or having a new lower level "Main" branch causes team confusion then name the new root branch "Trunk".

2. Reparent the Dev branches to the new root Main branch.
It is OK if the Dev branches remain in their existing TFS Projects, especially if there are many corresponding work items. The team could also decide when it is best to move the project if consolidation is desired, but that move can be separate from reparenting to a shared Main. TFS Projects are separate for administration and work items, but only appear separate for source code. Projects can easily have branches from other projects.
(Nested branches are frowned upon in TFS, but are doable if you turn off Branch Folder. That's likely another separate discussion)

3. Keep the original Main and Release branches for archive (since you have shipped previous releases through them).

4. If a hotfix is made to an existing release then either reparent that release to the new root Main branch OR discuss whether a baseless merge to root Main is better.
(CAUTION: I haven't walked this hotfix step through fully.)
...

I'll redraw my branches in a separate post. Feel free to reply to Q1 and/or my draft answer.


 

 

Developer
Feb 24, 2011 at 9:16 AM

In answer to your question. Let me start by saying I believ the answer should be not necessarily or at least not in all cases. This is my instinct. Now I will try to generalize some guidelines to support this.

With respect to your extrapolation, where you use Microsoft Excel as an example, I agree that Excel is a feature of office at a high level. In simplistic terms, you are correct that Word, Excel, Access could all be viewed as being developed as features of a single Main branch. In reality the branching structure is likely to be much more complicated with the Office team, but let's not let that complicate your basic point. If you think of Main as being the branch where vNext is stabilized prior to release it would seem to make sense that Excel, Word, PowerPoint, and so on, would share the same Main branch.

As for your Generalization, I am not sure that sharing source code is the correct factor for determining whether two or more projects should share a common *root* Main branch. The reason that Excel and Word share a common *root* Main branch is becuase they are features of the same product line and this product line ships as a single release (all features are part of the Office release). Even if Word and Excel did NOT share ANY common code, it would still make sense for them to be developed as features off of the same *root* Main branch.

I like to use another example of shared code, the Microsoft Enterprise Library, that was developed and released by the Patterns and Practices Group. Another example might be the .NET framework itself. You could have several unrelated projects that share code from Enterprise Library or that are built on top of the .NET framework. These projects could be developed entirely independently of each other. Think of Office and Visual Studio as a very general example. Office and Visual Studio do NOT share a common or *root* Main branch. These two product lines are completely independent of each other in terms of release cadence. They are on entirely different release schedules. Although there may be integration points or shared code between Office and Visual Studio, that does not argue in favor of having them under a common *root* Main branch.

Enterprise Library is a product unto itself. It is developed and released by Patterns and Practices. Its release cadence is separate from Office and Visual Studio, and for that matter separate from the .NET Framework. As such, it is not a feature of any of these product lines, but is a product line of its own. When you build your own .NET application, you could choose to build it with components of the Enterprise Library. You might have several projects that all use the same components from Enterprise Library. However, you would not necessarily release a new version of Enterprise Library simply becuase you are releasing a new version of a product that *uses* Enterprise Library components during development.  Because Enterprise Library and your various products are on different release schedules (and owned by different teams), they should NOT share a common *root* Main.

You have choices as to how you integrate components from Enterprise Library into your own projects (or products). You could share source code or you could share assemblies. When you share assemblies, it is clear that Enterprise Library and your products are not features of the same *uber* product. When you share source code the separation becomes somewhat less clear. For example you could add an Enterprise Library project to your solution as a project reference and incorporate the source code into your project. One benefit of doing this is you have the ability to extend Enterprise Library for your project (by inheriting from a base class and specializing it, for example).

In the Feb 2011 issue of MSDN magazine, I wrote an article that discussed various options for sharing code across projects. I would consider putting Enterprise Library into its own Team project with it's own Main branch. Then I would branch this Main branch of Enterprise Library into a folder with my various Team Projects that represent my products or product lines. Note that this Enterprise Library branch would not be a child branch of my product Main branch, but would simply be another branch within my product's Team Projects. This approach alllow me to change Enterprise Library within the context of my product and to have those changes kept separate (isolated) from the core source of Enterprise Library itself. If I had ten products, each with their own Main branch in their own Team Projects, I would branch Enterprise Library ten times - once into each product Team Project.

Having branched Enterprise Library into my product Team Project, I can easily add a project reference to the various Enterprise Library source projects. I could make changes directly to the source in these EntLib projects or I could derive new classes from existing base classes in EntLib and avoid making changes directly to EntLib source.

In any event, if I decide to make a change to EntLib, I can choose whether or not to share this change with other products in their various Team Projects. I have isolation of changes if I wish. But I also have a way of integrating changes from one user of EntLib to another. I would merge my changes to EntLIb from my product Team Project back to (RI) the Main Entlib branch in the Entlib Team Project. From there, I could merge these changes (FI) into the other product Team Projects I wish to share these changes with. The fact that many products share code in this way does not argue in favor of all my products sharing a common *root* Main branch. In essence I am suggesting the Main branch of the Shared Code Team Project be branched into the dependent Team Projects and then *treated* as a feature branch of the dependent projects. The EntLib branch is *treated* like a feature branch although it is not actually a child branch of the Main branch of any of the dependent products.

Bottom line... it is release coordination rather than code sharing that dictates for me whether two projects are features of a larger project and should therefore share a common *root* Main folder.

Regards,
Bill Heys
VS ALM Ranger

Developer
Feb 24, 2011 at 11:58 AM

A couple of additional comments regarding your Process. You state "Nested branches are frowned upon in TFS, but are doable if you turn off Branch Folder". Nested branches in TFS 2010 are not only frowned upon, they are not allowed. You will get an error if you convert a folder to a branch that is either nested under an existing branch or has already a nested branch. As you note, you can convert branches to folders or, during a branch operation, deselect the option of automatically converting folders to branches. In TFS 2010 you can have folders, branched folders, or first-class branches. The only restriction is that first-class branches cannot be nested under another first-class branch. No such restriction exists for branched folders, which function similarly to branched folders in TFS 2008. The primary benefit of first-class branches is branch visualization in TFS 2010.

The decision to organize visual studio solutions and projects into separate TFS Team Projects (I am careful to explicitly state Team Project) can be quite complex. There are often organizational considerations involving security. There are considerations around Process Templatest (a Team Project has only one Process Template). There are considerations around customizing Process Templates (WIT customizations, WI workflow customizations, Custom reports and Queries) - these are not shared implicitly across Team Projects.

Consoldating separate Team Projects into one Team Project, or consolidating Team Projects from mutiple Team Project Collections into one TP Collection is also not easy. The TFS Integration Platform can be used to move a Team Project, but such a move is not without some form of data loss (date compression and work item ID changes).

It is relatively easy, however, to split an existing TP Collection - simply by cloning the TPC and deleting Team Projects from one collection or the other.

Finally you mention choosing between reparenting a branch or doing a baseless merge. It is important to note that you cannot reparent a branch without first creating a merge relationship between the child branch and the new parent branch. This is done with a baseless merge.

Regards,
Bill Heys
VS ALM Ranger

 

Feb 25, 2011 at 1:29 AM

Hi Bill,

Thanks agin for your continued input. (Also thanks for the terminology correction regarding "TFS Team Projects" instead of "TFS Projects".)

EntLib (Enterprise Library) example - excellent discussion and exactly the type of feedback I was looking for. Thank you. We have recently adopted EntLib for our logging solution for all current and future projects. This means we now have a baseline sample implementation of custom EntLib project that will be significantly refined and extended in at least two parallel projects. This is where I was going with shared source code. I've read about making branches from a shared code TFS Team Project to consuming projects before (including reading your recent MSDN article :-)). I refer to these as "bridge branches" because they appear to connect two isolated branch trees "islands" (though consuming project just has the child branch in an adjacent folder, not actually connected to it's branch structure). That's part of where I got the idea of islands (and bridges) vs. a common Main(land) connecting the projects. :-).

The decision to organize VS solutions into separate TFS Team Projects was actually very easy for my team - because not much thought was originally put into having each major project have it's own TFS Team Project over 3 years ago and has continued to this day (though now with much more thought about admin & report implications). I'm confident many other companies have fallen into this pattern before learning there was any alternative pattern for project creation (each with it's own pros and cons). Consolidating TFS Team Projects is a much bigger topic (for another thread).

Baseless merge prerequisite of reparenting - good to know. Thanks! This also helps me think more carefully about the work and risks involved in reparenting several project branches to a new root Main... especially if any of those projects already have folders of the same name with different contents (diffirent version and/or functionality). One example I know of is the \3rdParty\... folder where we've stored third party binaries required to build our solutions. We may have collisions if any projects have same \3rdParty folder structure. Therefore even if it would have been better to share a common root main we might now find it unfeasible to retrofit this structure.

 

>Bottom line... it is release coordination rather than code sharing that dictates for me whether two projects are features of a larger project and should therefore share a common *root* Main folder.
I'd like to explore this further. Our team is developing SOA (Service Oriented Architecture) based solutions. This means most projects rely on many common services.

(Sorry the naming convention in the example below isn't very clever. ServiceAA1 = Service in Team Project "A" named "A" Version 1)
ProductA1 = WebsiteAA1+ServiceAA1+ServiceAB1+ServiceAC1
ProductB1 = WebsiteBA1+ServiceAC2 (from Team Project A with source code changes to support Product B) +ServiceBA1+ServiceBB1+ServiceBC1+ServiceBD1. Depends on ServiceAB1 already in Production.
ProductC1 = WebsiteBA2+WebsiteCA1+ServiceAC3 (from Team Project A with changes) +ServiceCA1+ServiceCB1+ServiceCC1+ServiceCD1+ServiceCE1+ServiceCF1... Depends on ServiceAA1 and ServiceBA1 already in Production.
ProductE1 = ...
ProductF1 = ...
ProductG1 = ...
ProductH1 = ...
etc...
ProductA2 = WebsiteAA2+ServiceAA1+ServiceAB2 (inherit changes from released ProductB1) + ServiceAC3+ServiceCB2+(incorporating and extending services from released ProductB1)
etc...

Services include file downloaders, file processors, application launchers, authentication, authorization, Business Logic service, Data Access Layer services (several), and much more.
CONCERN: You can see how branching for above scenario can become a serious issue if every service or every Product has it's own set of isolated branches. However that might be where we need to be.

Often new services have been developed in the TFS Team Project that initially creates the new service it needs. If we leave these shared services in their original Team Projects then the bridge branches to dependent projects will make diagrams look more like a hedge of bushes than a set of trees. (I don't think VSTS Branch Visualization within a Team Project will even show bridge branches by default (but can view all branches then hide all the truly unrelated ones.) Moving these services to a SharedService TFS Team Project would make branching structure more understandable (at least visually) but would also be work and requires losing easy access to historic work items (or doing a copy process).

Going forward we plan on developing shared services within a single Team Project with a common root Main branch. Only some of these services will ship for any given release, but they will all share EntLib and other source code and 3rd party binaries.

Currently we release ProductA by building all the related components in Team Project A and selected related solutions for services in Team Project B, Team Project C, etc. where we developed the changes needed for these services to support ProductA. We could branch ServiceAB to Team Project B.

Q: How much should we try to consolidate these 7 (or 20+) separte isolated Team Projects?

Using current guidance from this discussion it sounds like we should just create branch bridges whenever we need to change a service in isolation from its current version (and other potential parallel efforts). Parallel efforts on same shared component should communicate frequently and if changing same areas of code then do so in only one child branch (or do frequent merges through the parent service branch.

Q: How do we create branches for a specific service without nesting if that service shares a main branch?
I believe we'll need to branch the entire SharedServices Main to a folder in the consuming Team Project and then just modify/use the individual service(s) we need in that Project. We can use shared Mappings to cloak unused folders to simplify and reduce size of source code gets. Any better ideas?

I've got more thoughts, but they are not solid enough to write yet. With luck the above ramblings are coherent enough to be productive to all readers (not just myself).

Developer
Feb 25, 2011 at 1:46 AM

I sometimes get the impression that we are in a typing contest. What is our word count up to now? :)

Bill

Developer
Feb 25, 2011 at 2:19 AM

It is getting late at night, and I haven't been able to completely absorb you latest post. But I wanted to offer some thoughts as as I continue to ponder. Some of my suggestions are intuitive, some are based on pragmatic experience. Some are experimental, some are based on a set of core values. Bottom line, I may not have the perfect answer right off the top, but I am willing to push, probe, and evolve "our" thinking here. So my thoughts for tonight:

One of the tenents of SOA is that services can be versioned independently from other services (As I am sure you know well, a significatn challenge with services is when you change the interface). The services may be shared, but does that mean that they are shared code? Since shared services are decoupled from their clients, they can be developed and deployed independently from their clients (and other services).

Your comment about reparenting (We may have collisions if any projects have same \3rdParty folder structure) provoked some additional thinking about the feasibilty of trying to reparent many independently developed solutions under the same *uber* Main branch. I am uncertain what, if any, benefits you would realize by doing the reparenting.

I have also been dealing with a customer that would like to consolidate multiple Team Project Collections into a single Team Project Collection. The benefit of doing this from the customer's perspective is the ability to reporting across Team Projects which is feasible if these Team Projecs are all contained in the same Team Project Collection. The difficulty in doing this is there is no good way to move a Team Project from one TPC to another. If you use, for example, the TFS Integration Platform to do this, you do not end up with a full fidelity copy. There is a compression of dates (work item history implications) and a renumbering of work item ids, etc.

I would be inclined to say don't try to consolidate Team Projects unless there is a clear benefit. Your approach of *bridge* branches between Team Projects is an approach I have recommended in the past. The example I have used is this: Let's say that I have a product line (for example Microsoft Office) that I want to release. But before now, the individual products in this product line (Word, Excel, PowerPoint) were developed historically as independent products. Each product has its own Team Project. But let's say I want to integrate these products, begin to share more components (at the source code level) and release the new product line as a new unified release. One approach would be to create bridge branches from each of the products into a new product line team project. I am still evolving my thinking here, but I *think* that dependencies between these products will require (or at least suggest) that they be branched under a unifying Main branch in the unified Team Project. This is one technique for sharing code. The issue here is that when you add references to a Visual Studio Project, they are stored as relative paths. What happens if you have a Team Project $Office and you branch Word into this Team Project $Word\Main -> $Office\Word\Main. Then you branch Excel into this Team Project $\Excel\Main -> $Office\Excel\Main. Now you have a Main branch for the unified solution in the Office Team Project ($\Office\Main). Under the Main branch you create a unified Visual Studio solution. From there you add projects that reference projects under the Word or Excel folder. Now what happens when you branch Main to Development ($\Office\Main -> $Office\Main\Development\Feature Team1). Suddenly the relative paths in the target branch are different (nested deeper) than the relative paths in the source branch. Your project referecnes are now broken.

One solution is to branch your *bridge* branches under the Main branch in the Office Team Project. As Main is branched, all of the nested bridge branches are branched (this would require that the bridge branches be branched folders and not branches). However you structure this, you need to ensure that your relative path dependencies do not break when branched AND that your builds are not broken.

Just some thoughts for now.

Regards,
Bill Heys
VS ALM Ranger

 

 

Feb 25, 2011 at 5:06 AM

Thanks for your continue thoughts.  Your last post is exactly where I wanted this discussion to go. I appologize for the extra words and thoughts it has taken to get us this far. That said I am enjoying the journey and you have already helped me avoid several pitfalls. Sometimes I write a bit more explanation or mention caveats in hopes it helps others reading this to understand why we're not following an alternative otherwise left unmentioned. (* Anyone reading this is free to ask for shorter explanations :-D.*) It will be much easier to go back write a consice answer once the journey has reached a viable destination. At the end of this discussion I hope there is some good input for the next Branching Guide regarding SOA project challenges. (Perhaps there could be an example illustrating branching for 3 projects each consuming 1 or 6 of ~9 in-house shared services that also under development).

Team Project Consolidation: We are not looking to consolidate legacy projects. We are looking to find good pattern to follow when creating new services. For now I think our Platform Team Project will become the home to more services over time. If we need a separate workflow then we'll consider if using a $/Common Team Project is a better fit. I agree consolidating existing Team Projects must have clear team need that outweighs the time, risk, and compromises incurred by moving projects. (Two examples are cross-project reporting requirement or consolidated TFS Team Project administration requirement.) In the past I've seen a team pick a good time to make a clean break, then just branch the Project main to be in the new folder (and lock the old Team Project) or if necessary doing a "snapshot migration" and call the new branch the official home going forward. Someday there will be an easy tool to consolidate Team Projects... but not today.

Shared Services vs. Shared Source Code:
>The services may be shared, but does that mean that they are shared code?
Not normally, especially if the service is well established. However a V1 service often serves the original client well and is designed (hopes) to serve other clients in the future. When the next client (ProjectB) tries using the service there is often some development required to meet unforseen needs (or unmasked bugs). That is why I see new services needing to accomodate source code changes for the first three clients that consume it. (Hopefully no interface changes. Interface change means V1 service must remain in Production until first client updates to the newer contract.

Relative path preservation is good topic. I'll try commenting tomorrow on it (if nobody beats me to it :-)).

Cheers! -Zephan