Incremental Installer Design

Linden Lab Improved Installer/Updater

The goal of the installer is to maintain, upgrade or downgrade a copy of a specific runtime tree on local disk using the smallest number of downloads and least amount of local storage.

Runtime trees are hierarchically classified by:

 vendor/
   |_product/
      |_channel/

Only one active runtime tree per channel may exist. The creation of a new runtime should use the least amount of downloads and copy operations as possible.

Runtime trees are also implicitly classified by architecture. A seperate runtime tree exists for every supported architecture.

The benefits of this new architecture are:

• Faster updates
• Background updates
• Simplified re-branding, all version information can be removed from the actual files and specified in the index files instead, allowing reuse of the exact same binaries in different channels.
• Architecture independent process

Basic Idea

To store all files as opaque objects in a flat directory tree called the "staging area". Released runtime trees are represented by index files which map the opaque file to a relative runtime tree locations.

The staging area is just a flat collection of files, named according to they hexified SHA1 signature. The authoritative staging area lives on S3, with all files ever published. The client installer keeps a subset of the S3 staging area on local disk, with all files ever downloaded.

Patch Extension

Add patch files to the staging area which can be used to create one opaque object from an existing opaque object. Patch files are named <src-SHA1-sig>.<dst-SHA1-sig>. The index files will list all availabel patch files at the time the index file was created. The installer will then attempt to construct a requested opaque object using the available patch files.

Link Extension

Add link files which refer to the locations in the runtime trees where the original opaque object was copied. This reduces storage requirements for the staging area.

Components

The installer has two components:

The indexer

Takes a model runtime tree produced by our build system, and generate the appropriate index file and upload the required opaque objects to S3.

• In the patch extension, it will use the index file of a previous version to generate the patch files for those opaque objects which changed in the new version.

The updater

Polls for new index files applicable to the selected vendors, products, channel and the architecture of the platform running the installer. Depending on metadata provided in the index file and user preferences, one of the following actions will be performed:

• The index file is ignored
• The index file is processed and the required opaque object files are downloaded or constructed and made available as an optional upgrade on start of the next session
• as above, but no choice is given
• as above, but no choice is given and currently running processes are terminated.

The new runtime is constructed at startup time of the process by copying the opaque objects to their intended location accorting to the index file.

• In the Link extension, the opaque files are moved to their intended location and replaced by a text file with the same name as the opaque file, but with a .link extension appended. If an existing .link file exists, and the new runtime location differes from the location listed in the .link file, the new location is appended to the .link file.

Indexer Details

The indexer is run as part of the post build / packaging process. It will take as input:

• The vendor, product and channel name of the runtime
• A version id
• A path to the runtime tree
• In the patch extension, a path to a previous runtime tree for which the patch should be generated.

The indexer will produce:

• An index file mapping SHA1 hash signatures to relative runtime tree locations
• A set of opaque files to upload to S3.
• In the patch extension, a patch file for those opaque files which were in the same location in the previous runtime.
• The appropriate version manager entries.

The output will be produced by traversing the given runtime trees, computing the SHA1 signature for every file, and appending an entry to the index file. The indexer then issues a HEAD request to S3 to check if that file already exists, and uploads the file if it doesn't.

In the patch extension, binary diffs can be produced given a second runtime tree representing the previous version. The first step there is to compute the SHA1 signature from the reference runtime, followed by a HEAD request to see if the diff/patch file already exists. If not, then a new diff/patch file is generated and uploaded to S3. Finally, all the known patch file names are appended to the index file. This includes both the patch files generated in this pass, and all the patch files listed in the index file of the previous version.