Source version control

One of the problems in making large changes to the Second Life source is keeping track of changes, in order to be able to generate patches to submit. Also longer term, some way of keeping up with LL's changes to the source is needed. This page is an attempt to explain how to do that.

Ad-hoc

If you're not familiar with any source control systems, you can try to not use any, and instead just use the diff and patch utilities. This approach is very strongly discouraged because it doesn't scale at all. Nevertheless, it works OK for smaller things, like 1 line fixes to the source.

First, create a copy of the source:

% cp -Rdp linden linden.orig

Do some work on the linden directory. To make a patch use 'diff -u':

% diff -ru linden.orig linden > my_modification.patch

Keep the patch around. If LL releases a new version you can apply it to the new source:

% cd linden
% patch -p0 < ../my_modification.patch

This method has many disadvantages: it wastes huge amounts of disk space, is inefficient and error prone.

Subversion

Subversion is a much better option. However, it presents a bigger problem. LL offers a readonly Subversion repository that you can use to get the latest source. However, since you can't commit to it, you'll need to copy the source into your own repository. First, of all, you need to create your own repository. If you have several computers, you should do this on a machine you can reach from anywhere you might want to do development. Ideally it should be a machine accessible from the Internet.

For the purposes of the explanation, I will assume that your SVN repository is accessible through a web server at http://svn.example.com/secondlife. Check the Subversion manual for more information of how to access a repository.

Getting started

First, create a repository somewhere convenient. /var/lib is a common location for a server setup, but you can choose any other.

% svnadmin create /var/lib/svn/secondlife

Now create a local directory with the initial files. By convention, SVN repositories generally have "tags", "branches" and "trunk" directories at the top level. We won't create trunk yet. The main development happens in trunk. The 'vendor' directory is for holding the original SL source, with the latest version in vendor/current, and a tag in /vendor per SL release.

% mkdir tags
% mkdir branches
% mkdir vendor
% mkdir vendor/current

Import the basic structure into your SVN repository:

% svn import -m "Initial import" http://svn.example.com/secondlife

Now move the original directory out of the way, and check it out:

% svn co http://svn.example.com/secondlife

Importing the source

Now you have a working copy with the empty structure. Here comes the complicated part, how to integrate LL's source there and keep it up to date. Doing this is harder than it sounds, because you need to add new files, remove deleted ones, and keep track of renamed ones (if possible). The easiest way to do that is to use the svn-load-dirs utility.

First of all, get the SL source. Get the archive and extract it. I recommend checking in only the source package, and leaving the art and libraries out of the repository. The art is best left out because it's big and will slow things down if your SVN repository is located for instance on a shared hosting server. The libraries must be left out if you intend to let anybody else access your repository because some libraries in the archive can't be redistributed.

You should now have a "linden" directory with the source code. Now use svn-load-dirs to load it:

% svn-load-dirs -t 1.17.0.12 http://svn.example.com/vendor current linden

If your repository is accessed through the internet, this can take a while (even hours)

This will load the source from the linden directory, into http://svn.example.com/vendor/current, and at the same time create a tag in http://svn.example.com/vendor/1.17.0.12. The tag is a very good idea as it allows comparing LL releases easily, makes it easy to determine the last version you merged, and simplifies keeping up with LL releases a lot.

Now from your working copy, run "svn update":

% svn update

This will bring it up to date, and fetch the recently imported LL source. Now you can start your own branch:

% svn copy vendor/current trunk

Now you have the trunk branch for working in.

Merging LL releases

First, repeat the svn-load-dirs step above. Then you can merge the new changes into trunk:

% svn merge http://svn.example.com/vendor/<previously_merged_release> http://svn.example.com/vendor/current trunk

Now the new LL changes will be merged with your code. You can then "svn diff vendor/current trunk" to get a patch to submit.

SVK

The above method is workable, but it's not ideal, because SVN uses a centralized model. To use Subversion ideally you'd need to have commit access on the LL server. While it's possible you might get it, not everybody will, and you almost certainly won't get it immediately. Since you can't use the server directly, svn-load-dirs is a required hack. It's definitely not ideal. svn-load-dirs loses the history from the original server, and can't properly track renames (it will ask you, and you'll have to figure out somehow).

Distributed Development

Unlike systems like SVN and CVS, SVK is made for distributed development. In a distributed system there's no central server. The "official" server if there is such a thing is only official because people choose to believe it is, not because there's anything special about it. Whatever external servers you mirror with SVK are only accessed when you tell to SVK to sync with the server. The rest of the time, no network connection is needed, even for using commands like "svk log" and "svk blame" which would require a connection with SVN. This is possible because SVK mirrors the whole repository locally.

SVK makes it very convenient to keep in sync with external SVN servers. With just one command it'll fetch all the new revisions from a SVN server. Then you can do the actual merging and conflict resolution offline. Once you're done merging, you can push the changes to your own server. Another user could then use your server as a base for their code, and so on.

Smart Merge

The "smerge" or "sm" command is used very often when working with SVK. It's similar to the "svn merge" command in that it merges changes from one branch to another. However, "smerge" is much smarter than SVN's version. While the "svn merge" needs to be given a revision range or two branches so that it can compute what to merge, "svk smerge" automatically figures out, by finding the common ancestor of the branch you're merging from, and the branch you're merging to.

For example, suppose this scenario: We have the original source in /vendor/current, a branch /branches/foo, and trunk /trunk. Both /branches/foo and /trunk were created from /vendor/current at some point.

With SVN, we can merge the changes introduced in the "foo" branch into trunk like this:

% svn merge http://svn.example.com/vendor/current http://svn.example.com/branches/foo trunk

Now suppose that after the merge is done, somebody commits a new change to /branches/foo. With SVN we've now lost the ability to do an easy merge. Repeating the above command again would try to merge changes that were already merged, plus the newly committed changes. To correctly merge only the newly committed code, we must figure out what was the last revision we merged, and start from there.

SVK is much more clever in this respect. Here's how you'd do that:

% svk merge //project/branches/foo //project/trunk

Notice how we don't specify the vendor branch here. This is because SVK already knows what is the ancestor of both branches, and automatically figures out what to merge into trunk. Once you run this command, it'll record that as the last merge point. Repeating the command immediately would succeed without doing anything, because there aren't any new changes to merge. If something new is committed into //project/branches/foo, then only that would be merged.

This makes working with branches much easier in SVK. While with SVN you'd need to keep track of what you merged where, SVK does it for you.

Getting started

First of all, you need to mirror the SVN repository. This takes quite some time, but only needs to be done once. Run this command:

% svk mirror http://svn.secondlife.com/svn/linden //mirror/secondlife
% svk sync //mirror/secondlife

This will make SVK read the whole LL repository, with all its branches and revisions and store them on your hard disk (in your home directory).

When you're done, do the same to a repository you can write to (since you can't write to the LL one):

% svk mirror http://svn.example.com/sl //mirror/myproject
% svk sync //mirror/myproject

//mirror/secondlife and //mirror/myproject are special paths used by SVK. They're links to the actual repositories. Doing something to //mirror/myproject would result in changes being sent to your SVN server.

Creating the first branch

First, copy the branch you want to work with to your own server:

% svk mkdir //mirror/myproject/branches
% svk mkdir //mirror/myproject/branches/buildfixes
% svk smerge -B //mirror/secondlife/release //mirror/myproject/branches/buildfixes

"smerge" stands for "smart merge" and is a very cool thing. It intelligently merges starting from the part where you left. Since there's no common ancestor to both //mirror/secondlife/release and //mirror/myproject/branches/buildfixes (there can't be, since buildfixes is empty), you need to tell it to merge everything with the -B argument. After this initial step, further merges can be performed automatically.

Here we're creating a "buildfixes" branch on our own server, starting from the LL release source. I use the "buildfixes" branch as a starting point for all the others. It contains only the changes needed to get the source to build.

Creating local branches

A very good thing about SVK is that you can do useful work without a network connection. First, we'll create copies of the repositories for local use:

% svk copy //mirror/secondlife //ll/
% svk copy //mirror/myproject //myproject/

Changes made to these aren't automatically propagated to the SVN servers. They're remembered until you want to push them to the server. This means that you can sit on a plane without any sort of network connection, and still use the usual SVN commands -- check the log, commit, revert changes, etc.

Creating a new development branch

Now that the buildfixes branch is created, we can create another starting from it:

% svk mkdir //myproject/trunk
% svk smerge -B //myproject/branches/buildfixes //myproject/trunk

Notice how this time there won't be any network access. To push the changes to your server, do:

% svk smerge //myproject //mirror/myproject

Now suppose you updated the buildfixes branch, and want to propagate the changes. It's simple:

% svk smerge //myproject/branches/buildfixes //myproject/trunk

svk smerge keeps track of what was the last merge, and it automatically merges changes that weren't merged yet. Unlike with SVN, you don't need to keep track of revision numbers of make tags, it automatically does the right thing for you.

Merging new LL code

Now suppose LL releases a new revision and you want to merge it with your changes. This is easy:

% svk sync //mirror/secondlife
% svk smerge //mirror/secondlife //ll
% svk smerge //mirror/secondlife //myproject/branches/buildfixes

Since the buildfixes branch changed, you can propagate the changes further to the trunk branch:

% svk smerge //myproject/branches/buildfixes //myproject/trunk

And push it to your SVN server:

% svk smerge //myproject //mirror/myproject