Viewer Integration and Release Processes

This page describes how Viewer development projects are organized, including the Mercurial (hg) repositories, builds, and the integration and release processes through which changes move to get from the developer (either inside or outside Linden Lab) to the Second Life Viewer.

There are three core hg repositories (each repository is in effect a branch; the processes described here do not rely on using the hg branching features within a repository):

• Development is the integration trunk
• Beta is used to test and stabilize releases and build beta releases
• Release is used to build official stable releases

all three of these repositories are used by both Linden Lab development teams and open source developers; they are accessible at hg.secondlife.com, but only the specific Linden team responsible for each one has commit access to it. Detailed descriptions of each, and the processes for moving changes between them, are below.

Repositories

Development

Location: http://hg.secondlife.com/viewer-development

This is the conceptual ‘trunk’ from which all development, both inside and outside Linden Lab, should be derived.

It is built continuously, with the resulting viewers being publicly available from Downloading test builds and the Snowstorm Team page.

Any active viewer development, whether for feature development or bug fixes, pulls from (clones) this repository to a Project repository, and tracks all changes made in Development as work is done in that Project repository. Code is pulled back to this repository for integration: see Development Integration Criteria below.

Ideally, any build from this repository is a candidate for the next Beta viewer release.

Project

Location: determined by the development team, but somewhere at hg.secondlife.com or bitbucket.org is encouraged

Project repositories may be built as needed by the developers; viewers produced from Project repositories may be made available for public testing at times chosen by the development team.

Any development, whether a single engineer fixing a minor bug or a large team building a major feature, is done in a Project repository that pulls from Development. Large projects will likely have individual repositories pulling from and pushing to their team Project repository; how this is managed and tracks changes made to Development is under the control of the developer(s) on the project.

There may be any number of other repositories created by project teams or developers either inside or outside Linden Lab; these are Project repositories. The default channel identifier built from these repositories is "Second Life Developer". Projects that plan to make test viewers publicly available, which is encouraged, should change this to a project-specific value in those viewers.

When development is complete and the Development Integration Criteria have been satisfied changes from a Project repository are moved back to the Development repository. See How To Submit A Viewer Change.

Note: Developers should consider the viewer-development branch to be a "clean" trunk. It's not "pristine" -- we don't gate checkins through a QA process. But developers should feel very confident before checking in to viewer-development, because other projects, both internal and external, are pulling from this repository on a daily basis. They should feel confident in doing so, as it may require work to merge the changes. Consequently, developers must be willing to support their checkins.

If you are not prepared to respond to bug reports on a high-priority basis all the way through the release process, you should not check in.

Beta and Release

To create the Beta and Release Viewers, the changes on the Development repository are moved through the Beta and Release repositories following the Beta and Release Process.

These are continuous processes - soon (preferably within a week) after one set of changes has been moved from Beta to Release, the next Beta cycle is begun by moving the current Development changes to Beta to begin the next cycle. The number of weekly Beta cycles, and thus the frequency of windows for moving from Development to Beta, is determined by the Beta test results.

The targets are that we will have weekly Beta releases, with stable main viewer releases monthly.

Location: http://hg.secondlife.com/viewer-beta

The Integration and QA team tags Development and pulls from that tag into the Beta repository (since the Beta repository will contain only change sets that are already in Development, this pull does not require any merges).

Final QA and stabilization are done using builds from the Beta repository; builds from this repository are released weekly on the “Second Life Beta Viewer” channel.

Any bug fixes on this repository are immediately pulled to the Development repository by the Snowstorm Team.

Location: http://hg.secondlife.com/viewer-release

When the Snowstorm Team decides that the Beta branch is ready for release (which we expect to be after 2-4 weeks of testing and stabilization), the Integration team pulls changes from Beta into the Release repository (this is the only path into the Release repository, so this does not require any merges), where it is tagged and built to produce the viewer released on the “Second Life Release” channel.

Development Integration Criteria

In order to be eligible to be pulled into the Development repository, the changes in a Project repository must satisfy all of the following criteria:

1. Functionality must have been reviewed and accepted by the Product Owner

   Review is based on the requirements/user stories defined at the beginning of a sprint, and then at the end of a sprint during Acceptance Testing. For Linden development teams, these checks are the normal sprint reviews, not an additional step. For open source developers, the Snowstorm Team will coordinate these reviews with the appropriate Linden reviewers.

2. Design and code must have been reviewed by competent reviewers.

   In this context, “competent” means appropriate subject matter experts for the code that is modified. Linden development teams determine for themselves who the competent reviewers are; for open source contributions, the Snowstorm Team will determine the appropriate reviews. We have a public Code Review Tool for this purpose.

   • If there are changes to code or protocols shared between the viewer and the simulator, those changes must include unit tests that at minimum validate that the behavior before and after the change is compatible. If that is not possible due to the nature of the change, then the change must include documentation on how the interfaces are changed, and the relevant simulator developers must review and approve them (for open source contributions, the Snowstorm Team will coordinate and facilitate these reviews).

3. There must be a test plan

   The test plan must describe in detail how the modified behavior can be validated and tested. We provide a handy test plan template to help make this trouble free.

4. The Project repository must have merged in the latest changes from Development.

   The results must be validated by building viewers for all platforms and doing at least minimal viewer testing. Breaking the Development build is considered bad practice and may incur karmic debt.

   • There must be Contribution Agreements on file from all contributors to the change if the Project repository contains changes made by non-Lindens.

When these criteria have been met, the development team may integrate into the Development repository; see How To Submit A Viewer Change.

If a checkin causes excessive instability or breaks customer experience in important ways, it may be reverted and the developer will have to fix the issues and re-integrate. We hope not to have to do this often, but we will make the reversion decision quickly, because reversion usually requires reverting all later checkins as well.

Beta and Release Process

The process by which the Development code is promoted to an official numbered Viewer release is:

1. The Integration and QA team tags Development at the chosen changeset, increments version numbers, and propogates these changes to both viewer-beta and viewer-development; specifically:

   hg clone ssh://hg@bitbucket.org/lindenlab/viewer-development promote-repo

   cd promote-repo

   hg tag -m "add initial beta tag for <revision>" <revision>-beta1

   hg push ssh://hg@bitbucket.org/lindenlab/viewer-beta

   edit indra/llcommon/llversionviewer.h - increment LL_VERSION_MINOR

   hg commit -m "increment viewer version number to <new-revision>"

   hg push

2. The Release Manager
   1. Builds branch on Beta (this was probably automatically triggered on the push)
   2. Creates a release document that includes test plan information and the list of all the changes included in the build (list of JIRAs, etc). Proper attention to detail during development should make this simple.
   3. Offer build to QA for testing and evaluation
3. In QA, the integration test team:
   1. Executes the set of test plans for all of the changes included
   2. Performs a general smoke test on overall functionality
   3. Identifies overall quality and level of risk for release
4. The Snowstorm Team evaluates quality, risk and feature set and decides whether or not to release the build as Beta. Options are:
   • Fail: this beta candidate is rejected and the process starts over at some different tag on Development (step 1).
   • Fix First: specific bugs / issues are identified that must be fixed before ship. These bugs are fixed in the Beta repository.
     • Fixes made to the Beta repository are merged to Development by the Snowstorm Team as soon as they are found to be complete.
   • Ship: the Beta is ready as-is.
5. Once a build is approved for ship, the Release team does its magic to get the build into the field as a Beta build on the “Second Life Beta Viewer” channel.
6. Once a Beta is in the field, we watch for new bugs, crash rate, performance, and other data. If the Beta needs further iteration, we might iterate multiple times on the Beta branch (return to step 4).
7. Once the Snowstorm Team decides the Beta is of sufficient quality to ship as an official viewer, the Release team:
   1. Tags the Beta branch
   2. Pulls from the tag on Beta to Release
   3. Tags and builds from the Release repository.
   4. Releases on the “Second Life Viewer” channel.

Example Release Calendar

The following illustrates how and on which days the various events described above occur as a change moves from Development to Release (all times are PDT). The contents of each release are shown as releases A, B, C, and D. For simplicity, any releases prior to A are not shown. In a complete calendar, all weeks would like like week 3 below.

Week	Day	Viewer Repositories
		Development	Pre-Beta	Beta	Pre-Release	Release
1	Monday	06:00 Version A Development → Beta Candidate
		Version B commits	Version A Beta Candidate Built
	Tuesday		Version A QA of Beta Candidate
	Wednesday
	Thursday		Promotion Meeting 12:00
			Version A Beta Candidate → Beta
	Friday			Version A Beta Test
	Saturday
	Sunday
2	Monday	06:00 Version B Development → Beta Candidate
		Version C commits	Version B Beta Candidate Built
	Tuesday		Version B QA of Beta Candidate
	Wednesday
	Thursday		Promotion Meeting 12:00	Version A Beta → Release Candidate
			Version B Beta Candidate → Beta		Version A Release Candidate Built
	Friday			Version B Beta Test
					Version A Release Candidate QA
	Saturday
	Sunday
3	Monday	06:00 Version C Development → Beta Candidate
		Version D commits	Version C Beta Candidate Built
	Tuesday		Version C QA of Beta Candidate		Version A Release Candidate → Release
	Wednesday
	Thursday		Promotion Meeting 12:00	Version B Beta → Release Candidate
			Version C Beta Candidate → Beta		Version B Release Candidate Built
	Friday			Version C Beta Test
					Version B Release Candidate QA
	Saturday
	Sunday