Patch Updater Design

The objective of this project is to improve the user experience by making Second Life updates as simple, quick, and hassle-free as possible. The update system should attempt to match or exceed the user-friendliness of auto-updating applications such as Firefox. In particular, the project aims to eliminate the installation wizard part of the current updater, increase the reliability of updates, and reduce the download size. The tracking issue for this project is http://jira.secondlife.com/browse/VWR-603 . In the long-term, users will want close control over the version of their client -- they won't necessarily want to always update to the latest and greatest version.

Terminology

Installer

executable program that installs an application (Windows), .dmg file on Mac, or tarballs on Linux.

Update (n)

A file that contains newer code for your application. Commonly referred to as a 'patch'. But it may simply contain all the files for the newer version of the application.

Patch

An update that contains only the files for the newer version of the application that differ from the current version.

Updater

The application that applies updates to your application.

Unpacked

Linden-internal representation of the files in a complete install for a particular platform. Basically just a tarball of the files that the installer installs. The Linux installer already is a tarball, so the unpacked for linux would be just that. For Windows the installer is totally binary so we have to generate the unpacked separately.

How It Works Today

• When the viewer attempts to log in, it sends along its channel and version information to login.cgi, which checks to see whether there is a newer version in that channel.
• If there is an optional update, login.cgi rejects the login, sending back a code indicating that an optional update is available, and a text message describing the new version.
• When the user clicks "download", the viewer constructs a url to http://secondlife.com/update.php with a query string indicating the operating system and the grid of the viewer, and launches the updater process with this url as an argument.
• The updater fetches the url. Update.php uses the arguments in the query string to return a redirect to a file on S3 to download. The updater follows the redirect and downloads the file. It provides a rudimentary dialog showing download progress.
• After downloading the file, the updater applies it. On Windows this simply means executing the downloaded installer. On the Mac, the updater opens up the .dmg it has downloaded and copies the application inside over the old version's application bundle.

This process is two-step:

1. being notified that there is a new version
2. finding the file that updates your install to the newest version

Unfortunately, because the login servers are not always in sync with update.php, sometimes the login server will suggest an optional update, but when the updater downloads from update.php, it gets an older version, which causes the process to repeat until both servers sync up again. This could be cured if the updater knew which version it was supposed to download.

The other problem is that Residents cannot choose which version they wish to upgrade to. Institutional users quite often wish to use a version other than the latest, for stability, compatibility, and educational reasons. We should work to support this use case.

Example:

The ability to postpone a non-critical update is something many RL educators have been asking for. Many of them work with students in computer labs where general users do not have the ability to install/update software on the workstations, so updating SL is non-trivial for these folks. Also, RL educators frequently give presentations involving running SL, and a forced non-critical update tends to disrupt things. The ability to postpone non-critical updates would be a big win for the workflow of RL educators and students.

Improved Update Experience

Before even getting to patches, we have many service and UI changes that will dramatically improve the update experience for Residents. They are also (mostly) prerequisites to having a well-oiled and working patcher.

Background downloading is kind of a tricky nut to crack, and it's not a prerequisite, but it's in this section because of its conceptual relationship to the other two improvements.

Download Updates Via Web Service

The updater should be a very small application with a very limited scope of operation and dependencies, because its task is both so infrequent and so critical. The only reason it needs to be a separate application at all is because it isn't possible for a running process to overwrite its own executable.

One major problem with the current design is that the updater application does all the downloading. To do so while still remaining small, it uses native platform APIs that sort of do what we want, but are not very good at what they do. It doesn't have access to any string or url parsing libraries, and implementing more than very basic error checking would be difficult and counter-productive to do using the basic C functions. For example, the updaters cannot tell if the url they are given is a 404 or other error page, so instead they check the size of the downloaded file, and if it's less than a megabyte, the updater assumes that the download failed somehow.

The viewer already has extensive and cross-platform http downloading libraries, and a robust UI and ability to interact with the user. It only makes sense to start downloading the actual update file through the viewer itself. Then the viewer could kick off the updater process and pass along the location of the downloaded file. The updater would thus be much smaller and more reliable.

The initial implementation would work much like the existing system -- viewer logs in, gets optional update notice, downloads while Resident waits, then restarts with the new version. The updater application would be shrunk way, way down. It would do its job in moments, and not even display a UI if it doesn't have to.

Check for Updates Via Web Service

Note: all urls in this section are speculative.

The next step is to change the viewer so that it checks to see if there are new versions available via a web service instead of via login.cgi. Version checking logic should be almost completely eliminated from login, though login should still reject logins from 'blocked' versions, because those will be blocked for compatibility or security reasons.

There are a few times that you might want to check for notification of updates:

• on viewer start
• on login
• periodically while the viewer is running (logged in or not)

This should closely follow the design of Firefox's sucessful update notification scheme, documented at [1].

The viewer will request a url that specifies a bunch of information about the viewer:

https://updates.secondlife.com/update/%CHANNEL%/%VERSION%/%PLATFORM%/

The fields will be filled out appropriately for the viewer's current status.

CHANNEL

This is a string that arbitrarily groups together sets of versions. The viewer will know its own channel. See Using Channels for more information.

VERSION

The current version of the program. If we decide to support downgrading of the viewer version, these will appear as 'updates' to the current version, and the updater will have to distinguish them from newer versions.

PLATFORM

A string describing the platform that the user is using. Currently we have only three platforms: Windows, MacOS, and Linux, but we may wish to make finer divisions in the future (e.g. MacOS-ppc, Win64). This gives us the flexibility of releasing updates for one platform only.

The notifier service will look at the version manager to determine what updates are available to the viewer, based on the information in the url. It will construct a response that lists the available updates (based on [2]):

 [
   { 'version':'1.19.0.5',
     'channel':'Second Life Release',
     'type':'optional',
     'release_notes':'http://secondlife.com/app/releasenotes/1.19.0.5.win.html',
     'updates':{
       'installer':{
         'url':'http://secondlife.com/app/installer/SecondLife_Setup_1_19_0_5.exe',
         'size':'1034989',
         'checksum':{'type':'md5', 'value':'322352ab23'}
       }
     }
   },
   { 'version':'1.19.0.3',
     'channel':'Second Life Release',
     'type':'optional',
     'release_notes':'http://secondlife.com/app/releasenotes/1.19.0.3.win.html',
     'updates':{
       'installer':{
         'url':'http://secondlife.com/app/installer/SecondLife_Setup_1_19_0_3.exe',
         'size':'1033225',
         'checksum':{'type':'md5', 'value':'09a798b78e'}
       }
     }
   }
 ]

This response is simply a repackaging and summarization of the data in the version manager. This response would be highly cacheable, since it would only change (for a given url) when a new release came out and was added to the head of the list.

Note that the updates map contains only one entry: 'installer'. This would be to achieve parity with the current situation -- the updater downloads the installer and installs it! However, in the future it will be easy to extend the updates map with an additional entry: 'patch'.

The checksum entry inside the installer map allows the viewer to verify that the file downloaded correctly.

Size is in bytes.

Background Downloads

When you think about it, right when you want to log in is just about the worst time to be forced to watch a download progress bar go by on your screen while you do nothing. Once the updater is checking for updates via a web service, it becomes possible to download updates in the background, while the Resident is using Second Life. This is the same strategy as used by Firefox. Only when the application is started again is the update applied, ensuring that the interruption to the user's normal operation takes place at a time of his or her choosing, and also at a time when she was intending to restart the application anyway. The actual application of the update should be relatively rapid and thus won't interfere with the workflow as much.

We should try to match this model as best we can by streaming patch downloads alongside normal Second Life use. Clearly there is an issue of bandwidth contention here. Also there are questions about what to do when the user logs off and an update has only been partially downloaded. One situation that might be awkward is if the user downloads a patch behind the scenes and then doesn't use Second Life until after another update becomes available. When the user starts up again, she will see it install an update, then immediately start downloading another.

The Resident should always have the option to force an immediate download while offline.

NOTE: On Windows, we should consider using the [Background Intelligent Transfer Service http://msdn2.microsoft.com/en-us/library/Aa362827.aspx] which does automatic CPU and bandwidth throttling to make it have minimal impact on users.

Background UI Design

What follows are some notes about what UI elements would need to be implemented to make background downloads as transparent and useful to Residents as possible.

Update downloaded notification

This is the notification that pops up when the update has finished downloading. It should have three buttons:

• More Info (brings up update viewer)
• Restart and Install(restarts SL and installs update)
• Continue (dismisses notification)

Update notification

This is the message that pops up when an update check find new versions while the user is logged in and the user has asked to be notified.

This should be a notification with three buttons:

• More info
• Cancel/Postpone
• Download

More info brings up the updates viewer.

Updater progress bar

This progress indicator will appear when the user restarts the viewer after an update has been downloaded and is being applied. This won't show up at all if the update is an installer, because the installer provides its own dialog. It shouldn't appear for very long since the updates will be already downloaded and will simply need installation/patching. The Mozilla version of this progress bar only appears if the updates are taking longer than half a second to install.

It's simply a progress bar, and there's little point to displaying more information here (though I do think it's important to display the version and channel).

"Bug Me" Preferences

These preferences allow the user to control how annoying the auto-updates are. I've ripped them quite literally from the Firefox playbook. The only difference with us is that we sometimes have mandatory updates, so the viewer will always check for those.

• Notify me when updates have been downloaded.

The nice thing about checking if the user wants the updates prior to download is that we can always assume that any downloaded updates are wanted.

Update Status Floater

Allows the Resident to get information about a version that is available for download.

To add:

• Preferences button
• Download progress (%, kb, and bar)

The action buttons should differ depending on the type of update. Optional updates should have "Download" or "Postpone" actions. Mandatory updates should have the old standard, "Download" or "Quit". If the download is already started, there should be a "Stop Downloading" button.

From the user's perspective, mandatory updates should be detected no later than login. Optional updates have more flexibility in when they're checked for, though we might as well check for them at the same time as we're checking for mandatory updates.

If there is no update, the content will simply be "no updates available" and a "Check now" button.

Icon Notifier

We want to not bother the user with dialogs while an update is background downloading (unless they specifically request to be bothered), but we also want to notify them that a background download is taking place. Hence, I think it's worthwhile to provide an unobtrusive indicator that lets the Resident know when we're checking for updates when they're logged in, when we're downloading them, and when a download is ready for installation.

The following is a quick mockup of such an indicator. It's the little up-arrow.

To change: arrow should be circular, like a recycling icon

The indicator should visibly display the update status.

No activity

Arrow is not visible

Checking for available updates

Arrow is visible and completely black in color.

Downloading an update

Arrow fills with blue from the base to the tip as the file is downloaded. (shown)

Update downloaded

Arrow is solid green (distinguishable from the god-mode green)

Update postponed

Arrow is not visible

Clicking on the arrow brings up the Update Status Floater, showing information about the in-progress update.

Track Updates to the Updater

Keeping the updater's version tied to the viewer version causes The Fear for everyone who wants to change the updater. If the updater breaks somehow, then everyone who has a broken updater has to go to our site and download a newer version to get fixed. It should be possible to update the updater itself asynchronously from the main application so that when we publish fixes for the updater, they are downloaded by all viewers.

It's very simple for the viewer to update the updater -- it simply has to download the newest updater executable and copy it over the old updater executable. The viewer doesn't have to exit, and the updater executable should be very small so the entire process should be essentially invisible to the Resident.

To check for an update to the updater, the viewer needs to know enough about the updater to be able to generate an update URL for it. It will request this information from the updater application. E.g. it will run updater --version, which will output the LLSD:

{
  'version':'1.0',
  'channel':'Second Life Updater',
  'platform':'win'
}

The viewer will use this information to construct an url to the update web service, the same way as [#Check_for_Updates_Via_Web_Service the section above]. Based on the results of checking the web service, the viewer can simply download the latest updater and copy it over the existing updater executable.

Patching

Hey it's halfway down the page and we're finally getting to the patch part? Update logic is tricky, and it turns out that patches are a relatively small change relative to the UI changes of the previous sections. From the user perspective, the only differences a patch can make is that it takes less time to do stuff! Patches are also something of a risk since they can wreck the install directory, forcing the Resident to manually download an installer for the desired version.

Because of the layout of our install directory, where almost all of the data is in the single executable file, we have to use a binary delta-patching strategy. This strategy means that to generate the patch, we take two different versions, A and B, and run a script over them that generates a binary diff between them. A Resident with version A can then download the binary diff, and with the help of another tool (hidden in the updater, naturally), be patched up to version B. The diff does not work if applied to any version other than A, and it also will fail if the Resident has modified any of the files that are to be patched. So it's brittle, but each patch is also the minimal possible size.

Most of the patch-related changes are on the server side. Because we only patch between versions, we have to keep a copy of each released version on-hand to be able to generate these binary diffs to newer versions. Whenever a new version is released, we want to generate patches to the new version from some of the older ones. We probably want all this to be automatic, because it will be too annoying to do by hand.

There are three components to a patching system: modifications to the version manager to support patch notification, a way to generate patches and store them in the proper place in the version manager, and modifications to the updater application so that it can apply downloaded patches to the viewer.

Patch Notification

Once we are ready to start patching, the responses from the update notification web service should add two new types of update: patches, and full. The content would then look something like this:

[
   { 'version':'1.19.0.5',
     'channel':'Second Life Release',
     'type':'optional',
     'release_notes':'http://secondlife.com/app/releasenotes/1.19.0.5.win.html',
     'updates':{
       'installer':{
         'url':'http://secondlife.com/app/installer/SecondLife_Setup_1_19_0_5.exe',
         'size':'1034989',
         'checksum':{'type':'md5', 'value':'322352ab23'}
       }
       'patch':{
         'url':'http://secondlife.com/app/installer/1_19_0_4_to_1_19_0_5.win.patch',
         'size':'305856',
         'checksum':{'type':'md5', 'value':'12bdbab211'}
       }
     }
   }
 ]

Note the addition of the 'patch' entry. It's smaller.

Patch Generation

It's actually quite easy to generate a patch between two directories. All you gotta do is:

• Walk both directories, generating a list of files.
• Using set operations, create three lists of files: those that are found only in the source, those found only in the destination, and those common to both.
• Create a third directory for the patch, and copy every file found only in the destination into the patch directory, preserving hierarchy
• For every file in the set of common files:
  • Check md5s of source and destination files, and skip if they're the same
  • For differing files, run bsdiff binary to generate a binary patch between them, and store the resulting patchfile in the patch directory, preserving hierarchy
• Construct a manifest listing all the files in the sets, describing whether each file was added, removed, or patched (unchanged files can be skipped).
• Tar up the patch directory and manifest file.

Now you have a patch!

There is some uncertainty about which compressed archive format should be used in the last step. I suspect that the decision will hinge upon which of these is easiest to integrate with the updater's code.

• libzip creates .zip files, the old standby
• libtar creates .tar files, the even older standby. We'd want to couple this with zlib so that it's actually compressible.
• If we're going to completely gank Mozilla's implementation, the pack will be in MAR format. MAR is an archive format designed to minimize the implementation code and dependencies, but it's not especially standard. However, it's not like you get any utility from unpacking the patch anyway (cause it's full of binary diffs).

The combination of different compression algorithms tends to achieve higher overall compression, so since bsdiff uses bz compression, the transport should use something else.

Manifest File

The manifest file lives in a fixed location in the archive (update.manifest). It serves as both an index (since most of the compressed archive formats don't have one) and instructions on what operations to perform.

The manifest file looks like this:

add app_settings/panel_group_general.xml
patch SecondLife.exe.patch SecondLife.exe
remove SecondLife.lnk

Note that this is not LLSD. It could be, but then again the goal is to make the program which parses this file (the updater) be as simple as possible. A restricted-semantics text file is a great way to achieve this. Whatever's easiest to integrate and makes the updater smallest, really.

Each of these entries corresponds to an action that the updater can take.

add

Adds a new file, or replaces an existing one.

patch

Applies a binary patch to an existing file.

remove

Removes an existing file.

Each of these actions is reversible, in the case of error.

This is based off of the Mozilla code, but given that we can expect our users to make changes to the files in their local directories, we probably want to keep track of the expected checksum of the initial and final states of the files, like this:

add app_settings/panel_group_general.xml 3ba0987ddfd 
patch SecondLife.exe.patch SecondLife.exe b7856de765f ad98f7897b9
remove SecondLife.lnk a9f8e8023b2

The updater will verify the checksums of the files before and after each step, and will abort and rollback the update if any file fails.

Creating Unpackeds

An unpacked installation (or 'unpacked') is a package containing enough information to be able to generate a diff with any other version. These are the guys that we'll have to keep around for every version on every platform. Essentially an unpacked is the installation directory that the installer would create. The initial implementation will be just that -- each unpacked version will be a directory containing the files in the installation directory for each platform.

The build process for release can currently generate both an unpacked and an installer, on all relevant platforms. All you have to do is run viewer_manifest.py with --actions='copy unpacked package'. We should change the default actions so that it generates an unpacked every time.

Tracking unpackeds

We'll have to keep around unpackeds for a while so that we can make patches between them. I expect that every time we have a new release, we'll pick a set number of older unpackeds to generate patches to the latest.

Patch Creation Policy

Since our assumption is that all relevant information is contained in the unpacked, then the system which generates the updates should take unpackeds as its input.

The backend should then go about inserting the unpacked into storage and generating any relevant patches from older versions to the new release.

Updater Application

The updater application has the minimal amount of functionality. It will be run when the viewer starts and detects downloaded updates. The updater checks a specific location for updates, checks them for validity, and applies them.

Downloaded Updates

The downloaded updates will be in %SL_INSTALL%/updates/. In this folder will be the downloaded packs that have not been applied yet, and a file called updates.xml.

The contents of updates.xml will be very close to the response from the web service. However, updates.xml will only contain the sections relating to the packs that have been downloaded. E.g. in reference to the earlier example,

[
  { 'version':'1.19.0.5',
    'channel':'Second Life Release',
    'type':'optional',
    'release_notes':'http://secondlife.com/app/releasenotes/1.19.0.5.win.html',
    'updates':{
      'patch':{
        'url':'file://C:/Program%20Files/SecondLife/updates/Second_Life_Release-1_19_0_3-1_19_0_5.patch',
        'size':'305856',
        'checksum':{'type':'md5', 'value':'12bdbab211'}
      }
    }
  }
]

Note that the url has been changed to reference a local file with a path relative to the updates directory.

The updater applies the packs in the order of their appearance in the updates.xml file, assuming that they have the correct version. A smarter updater can skip straight to the first full pack or installer.

When applying each pack, the updater opens the archive, looks at the manifest, and performs the operations found therein.

Rollout strategy

Our rollout strategy should involve two stages once the code is ready:

1. Shipped but switched off, users can enable a setting that uses the patching updater. The updater sends us somewhat detailed trace logs detailing how it's doing.
2. Turned on by default. Users who haven't explicitly disabled the patch updater will use it. The old download-the-installer updater will still ship, but won't be used. Eventually we can eliminate it.

Research

How many patches?

This section is about how patches are aggregated for download -- as a series of incremental patches, or as multi-version delta.

Incremental patches

Each patch upgrades from one incremental version to the next highest version. If a user's viewer lags the desired version by several versions, he has to download and install all intermediate patches in succession. E.g. we would provide patches for 1.12.1.3->1.12.1.4, 1.12.1.4->1.12.1.5, 1.12.1.5->1.12.1.9, and so on.

Multiversioned patches

Each patch upgrades over a set of versions. If a user's viewer lags the desired by several versions, he downloads one patch that brings him completely up to date. E.g. he would download a 1.12.1.3->1.12.1.9 patch.

The advantage of incremental patches is that they are conceptually simple to execute, and take up very little room on our servers. The downside is that users who have to download more than one incremental will end up downloading more bytes total. In some cases, the size of the accumulated incrementals would exceed the size of downloading a new installer from scratch.

From a user perspective, multiversioned patches are clearly the ideal. They represent the minimum possible download to bring him up to date. The downside is that the server space will increase faster than the incremental strategy, especially given the use case of users not always wanting to update to the latest version. Keeping around patches to go from any particular version to any other version will grow at N², which is probably unsustainable.

The eventual strategy will be a hybrid one -- both the viewer and the patch server will understand and apply both incremental and multiversioned patches.

All strategies require that we keep around an unpacked for each version, for each platform on which it was released.

Prexisting updaters

The market seems to be focused on supporting IT management of Windows and Unix security patches. Turn-key solutions to create auto-updating software applications seem to be few and far between. Additionally, proprietary software isn't really an option for us anymore.

Sparkle

Cocoa-only, it seems. Would be worth examining to see if it could be easily made cross-platform.

Xdelta

Library and app for binary diff-ing. Major limitation is that it is rather bare-bones and immature.

Lindersoft SetupBuilder

The very fact that their site doesn't work in Firefox gives you some indication of how Windows-centric they are. They provide web services that manage patch sets, although I can't tell from their site what control you have over this, if any.

RTPatch

A comprehensive patch creation/management suite. Also requires that you use their installers. It looks like they support HP-UX but not OS X, which is strange.

AutoUpdate+

A glossy patch creation and management tool. It is Windows-only, and the patches that it generates run wizards, which is something we specifically wish to avoid.

Advanced Installer

You can generate patches if you switch your entire infrastructure over to Windows Installer. No patch management, though.

Shareware patch creation

I'm just going to lump the dozens of tiny patch creation tools that I found. They all look similar, and they all do the same thing, which is create a .exe that the user downloads and installs. If we wanted to do that, we'd just use VPatch.

Estimated Patch Size

Here are some estimates of how big the patch updates are likely to be. I used the following procedure to generate each patch:

• Get install directories for two successive versions
• Iterate over the files in both directories, comparing files via md5sum
• If a file differs between directories, compute a binary diff (using bsdiff), then stash the diff in an output directory tree. If the file is new, it is stashed whole.
• Zip up the patch directory tree.

This method is fairly close to the implementation I'd like to use, so the estimates are probably within about 30% of the actual future performance, and within an order of magnitude of the smallest patches theoretically achievable.

The time taken to generate diffs is nontrivial, and proportional to the size of the largest executable, which is the largest file in the install. For Linux, it takes ~800s; for Windows ~200s; for Mac, ~4300s (!). This is all timed on my G5, not the fastest box in the toolshed. Bsdiff appears to be memory-bound, because the CPU utilization is not high, nor is the disk thrashing, it's just allocating gigabytes of virtual and repeatedly scanning over them.

And here are the sizes:

	Windows	size (MB)	Mac	size (MB)	Linux	size (MB)
	1.12.0.14 to 1.12.0.15	.9	1.12.0.14 to 1.12.0.15	2.7
	1.12.0.15 to 1.12.1.6	2	1.12.0.15 to 1.12.1.6	12
	1.12.1.6 to 1.12.1.8	.9	1.12.1.6 to 1.12.1.8	2.4
	1.12.1.8 to 1.12.1.9	.9	1.12.1.8 to 1.12.1.9	3.8
	1.12.1.9 to 1.12.1.11	.8	1.12.1.9 to 1.12.1.11	4.6
	1.12.1.11 to 1.12.1.12	.6	1.12.1.11 to 1.12.1.12	5
	1.12.1.12 to 1.12.2.5	1.2	1.12.1.12 to 1.12.1.13	1.4
			1.12.1.13 to 1.12.2.5	5.5
	1.12.2.5 to 1.12.2.7	1.2	1.12.2.5 to 1.12.2.7	2.9
	1.12.2.7 to 1.12.2.8	.4	1.12.2.7 to 1.12.2.8	.8	1.12.2.7 to 1.12.2.9	10.8
	1.12.2.8 to 1.12.2.9	.6	1.12.2.8 to 1.12.2.9	.5
	1.12.2.9 to 1.12.3.4	1.5	1.12.2.9 to 1.12.3.4	5.8	1.12.2.9 to 1.12.4.1	8.6
	1.12.3.4 to 1.12.3.6	1.3	1.12.3.4 to 1.12.3.6	7.1
	1.12.3.6 to 1.13.0.8	2.4	1.12.3.6 to 1.13.0.8	8.1	1.12.4.1 to 1.13.0.8	4.2
	1.13.0.8 to 1.13.0.10	1.2	1.13.0.8 to 1.13.0.10	4.6	1.13.0.8 to 1.13.0.10	2.3
	1.13.0.10 to 1.13.1.5	8.4	1.13.0.10 to 1.13.1.5	13.4	1.13.0.10 to 1.13.1.5	10.6
Average	--	1.62 MB	--	5.0 MB	--	7.3 MB
Full size	--	22 MB	--	56 MB	--	34 MB
Proportion	--	7%	--	9%	--	21%

Note that 1.13.1.5 is the release with additional data in the VFS -- it is ~7 MB larger than previous releases across all platforms. The "full size" row reflects the pre-VFS-additions sizes.

Mozilla Links

• The updater is in Mozilla's source tree, in mozilla/toolkit/mozapps/update
• http://wiki.mozilla.org/Mozilla_QA_Community:Software_Update
• http://wiki.mozilla.org/Software_Update
• http://lxr.mozilla.org/mozilla/source/tools/update-packaging/
• http://lxr.mozilla.org/mozilla/source/tools/patcher (this seems rather similar to the previous)
• http://wiki.mozilla.org/Software_Update:Nightly_update_infrastructure