Linden Lab Improved Installer/Updater

The goal of the installer is to maintain, upgrade or downgrade a copy of a specific runtime tree on local disk using the smallest number of downloads and least amount of local storage.

Runtime trees are hierarchically classified by:

 vendor/
   |_product/
      |_channel/

Only one active runtime tree per channel may exist. The creation of a new runtime should use the least amount of downloads and copy operations as possible.

Runtime trees are also implicitly classified by architecture. A seperate runtime tree exists for every supported architecture.

The benefits of this new architecture are:

Faster updates
Background updates
Simplified re-branding, all version information can be removed from the actual files and specified in the index files instead, allowing reuse of the exact same binaries in different channels.
Architecture independent process

Basic Idea

To store all files as opaque objects in a flat directory tree called the "staging area". Released runtime trees are represented by index files which map the opaque file to a relative runtime tree locations.

The staging area is just a flat collection of files, named according to they hexified SHA1 signature. The authoritative staging area lives on S3, with all files ever published. The client installer keeps a subset of the S3 staging area on local disk, with all files ever downloaded.

Patch Extension

Add patch files to the staging area which can be used to create one opaque object from an existing opaque object. Patch files are named <src-SHA1-sig>.<dst-SHA1-sig>. The index files will list all availabel patch files at the time the index file was created. The installer will then attempt to construct a requested opaque object using the available patch files.

Link Extension

Add link files which refer to the locations in the runtime trees where the original opaque object was copied. This reduces storage requirements for the staging area.

Components

The installer has two components:

The indexer

Takes a model runtime tree produced by our build system, and generate the appropriate index file and upload the required opaque objects to S3.

In the patch extension, it will use the index file of a previous version to generate the patch files for those opaque objects which changed in the new version.

The updater

Polls for new index files applicable to the selected vendors, products, channel and the architecture of the platform running the installer. Depending on metadata provided in the index file and user preferences, one of the following actions will be performed:

The index file is ignored
The index file is processed and the required opaque object files are downloaded or constructed and made available as an optional upgrade on start of the next session
as above, but no choice is given
as above, but no choice is given and currently running processes are terminated.

The new runtime is constructed at startup time of the process by copying the opaque objects to their intended location accorting to the index file.

In the Link extension, the opaque files are moved to their intended location and replaced by a text file with the same name as the opaque file, but with a .link extension appended. If an existing .link file exists, and the new runtime location differes from the location listed in the .link file, the new location is appended to the .link file.

Indexer Details

The indexer is run as part of the post build / packaging process. It will take as input:

The vendor, product and channel name of the runtime
A version id
A path to the runtime tree
In the patch extension, a path to a previous runtime tree for which the patch should be generated.

The indexer will produce:

An index file mapping SHA1 hash signatures to relative runtime tree locations
A set of opaque files to upload to S3.
In the patch extension, a patch file for those opaque files which were in the same location in the previous runtime.
The appropriate version manager entries.

The output will be produced by traversing the given runtime trees, computing the SHA1 signature for every file, and appending an entry to the index file. The indexer then issues a HEAD request to S3 to check if that file already exists, and uploads the file if it doesn't.

In the patch extension, binary diffs can be produced given a second runtime tree representing the previous version. The first step there is to compute the SHA1 signature from the reference runtime, followed by a HEAD request to see if the diff/patch file already exists. If not, then a new diff/patch file is generated and uploaded to S3. Finally, all the known patch file names are appended to the index file. This includes both the patch files generated in this pass, and all the patch files listed in the index file of the previous version.

Updater Details

The updater is itself an installed application, installed via a native installer. The idea is to have this be the only architecture dependent portion of the whole process. The remainder of the code is written in python and should be platform independent.

The updater acts as an execution wrapper for the applications it manages.

The updater can be triggered in two ways:

A new version of the updater is downloaded via the Second Life download page
A running viewer triggers an upload cycle:
- At startup;
- In regular intervals while running;
- Upon receipt of a server side message.

The updater proceeds in three steps:

Download of a new index file;
Download of the missing opaque files from S3;
- in the patch extension, this may involve downloading patch files and constructing the opaque files from the patch files;
- in the link extension, the contents of opaque files may need to be retrieved from existing runtime trees. The verification of the availability of those files is performed here, and in case of a missing runtime file, the link file is replaced with a fresh download.
Construction of a new runtime tree. This step is performed on startup of a new viewer process. Depending on the urgency of the update, with or without confirmation by the user.

Index File Details

The index file is the bootstrap file for the download. It consists of two parts:

The header, containing metadata:
- vendor/product/channel;
- the command line and options used to launch the application;
- urgency:
  Critical
  
  Currently running viewer is shut down, update proceeds, runtime updated on startup without user confirmation;
  
  Mandatory
  
  Currently running viewer may continue to run, update proceeds, runtime updated on startup of next session without user confirmation;
  
  Optional
  
  Currently running viewer may continue to run, update proceeds, runtime updated on startup of next session only with user confirmation;
- new channel name - used to rename or merge channels. The runtime tree will be constructed using the new channel name, and the old channel will be removed.
- comment: Will be displayed in the notification or confirmation dialogue on startup of the new session.
The body, containing one line per file, with two entries each, separated by a space:
- SHA1 signature in hex;
- relative path in the runtime tree (may contain spaces).

The index file is downloaded in two ways:

On installation of the updater itself. The downloaded updater will have a vendor/product/channel triplet hardwired as the default.
Via background polling on viewer session startup, in which case the viewer provides the vendor/product/channel triplet to poll.

Download Details

Upon receipt of the index file, files not already present in the staging area (or constructible via patch files in the patch extension, or referenced via link files in the runtime trees in the link extension) will be downloaded from S3. This will have no effect on the active runtimes which can continue to be used while the downloads are happening. Since the runtime tree files and the staging area files are distinct, there are no conflicts between the applications and the updater.

If multiple instances or multiple applications are running in parallel, the updater must either serialize downloads or provide proper locking mechanisms to avoid conflicts within the staging area.

If the downloads complete successfully, the index file is placed in a defined location within the runtime of the application.

Runtime Construction Details

The runtime construction happens on session startup only. This is implemented by making the updater be the launcher for the applications it manages.

Upon launch, it checks for the existence of a new index file in a specific location within the runtime tree, and depending on the urgency setting, proceeds to update the runtime tree. Upon successful completion, the index file is moved to a different location for possible future use, for example integrity verifications or for optimizing the process of updating the next runtime, and the new version of the application is launched.

Incremental Installer Design

Contents