Source version control

One of the problems in making large changes to the Second Life source is keeping track of changes, in order to be able to generate patches to submit. Also longer term, some way of keeping up with LL's changes to the source is needed. This page is an attempt to explain how to do that.

Ad-hoc

If you're not familiar with any source control systems, you can try to not use any, and instead just use the diff and patch utilities. This approach is very strongly discouraged because it doesn't scale at all. Nevertheless, it works OK for smaller things, like 1 line fixes to the source.

First, create a copy of the source:

% cp -Rdp linden linden.orig

Do some work on the linden directory. To make a patch use 'diff -u':

% diff -ru linden.orig linden > my_modification.patch

Keep the patch around. If LL releases a new version you can apply it to the new source:

% cd linden
% patch -p0 < ../my_modification.patch

This method has many disadvantages: it wastes huge amounts of disk space, is inefficient and error prone.

Subversion

Subversion is a much better option. However, it presents a bigger problem. LL offers a readonly Subversion repository that you can use to get the latest source. However, since you can't commit to it, you'll need to copy the source into your own repository. First, of all, you need to create your own repository. If you have several computers, you should do this on a machine you can reach from anywhere you might want to do development. Ideally it should be a machine accessible from the Internet.

For the purposes of the explanation, I will assume that your SVN repository is accessible through a web server at http://svn.example.com/secondlife. Check the Subversion manual for more information of how to access a repository.

First, create a repository somewhere convenient. /var/lib is a common location for a server setup, but you can choose any other.

% svnadmin create /var/lib/svn/secondlife

Now create a local directory with the initial files. By convention, SVN repositories generally have "tags", "branches" and "trunk" directories at the top level. We won't create trunk yet. The main development happens in trunk. The 'vendor' directory is for holding the original SL source, with the latest version in vendor/current, and a tag in /vendor per SL release.

% mkdir tags
% mkdir branches
% mkdir vendor
% mkdir vendor/current

Import the basic structure into your SVN repository:

% svn import -m "Initial import" http://svn.example.com/secondlife

Now move the original directory out of the way, and check it out:

% svn co http://svn.example.com/secondlife

Now you have a working copy with the empty structure. Here comes the complicated part, how to integrate LL's source there and keep it up to date. Doing this is harder than it sounds, because you need to add new files, remove deleted ones, and keep track of renamed ones (if possible). The easiest way to do that is to use the svn-load-dirs utility.

First of all, get the SL source. Get the archive and extract it. I recommend checking in only the source package, and leaving the art and libraries out of the repository. The art is best left out because it's big and will slow things down if your SVN repository is located for instance on a shared hosting server. The libraries must be left out if you intend to let anybody else access your repository because some libraries in the archive can't be redistributed.

You should now have a "linden" directory with the source code. Now use svn-load-dirs to load it:

% svn-load-dirs -t 1.17.0.12 http://svn.example.com/vendor current linden

If your repository is accessed through the internet, this can take a while (even hours)

This will load the source from the linden directory, into http://svn.example.com/vendor/current, and at the same time create a tag in http://svn.example.com/vendor/1.17.0.12. The tag is a very good idea as it allows comparing LL releases easily, makes it easy to determine the last version you merged, and simplifies keeping up with LL releases a lot.

Now from your working copy, run "svn update":

% svn update

This will bring it up to date, and fetch the recently imported LL source. Now you can start your own branch:

% svn copy vendor/current trunk

Now you have the trunk branch for working in. Go and do something in there. Now suppose LL releases a new version. First, repeat the svn-load-dirs step above. Then you can merge the new changes into trunk:

% svn merge http://svn.example.com/vendor/<previously_merged_release> http://svn.example.com/vendor/current trunk

Now the new LL changes will be merged with your code. You can then "svn diff vendor/current trunk" to get a patch to submit.

SVK

The above method is workable, but it's not ideal, because SVN uses a centralized model. To use Subversion ideally you'd need to have commit access on the LL server. While it's possible you might get it, not everybody will, and you almost certainly won't get it immediately. Since you can't use the server directly, svn-load-dirs is a required hack. It's definitely not ideal. svn-load-dirs loses the history from the original server, and can't properly track renames (it will ask you, and you'll have to figure out somehow).

An alternative is SVK. SVK is a distributed source control system based on SVN. In a distributed system there's no central server. The "official" server if there is such a thing is only official because people choose to believe it is, not because there's anything special about it.

A crucial difference is that SVK can mirror the LL SVN repository, keeping all the data about revisions, with one command. Then you can make your changes to that, and push them to your own SVN server. If you make that available, somebody could mirror that, and so on. SVK makes it very convenient for people to create their own branches based on bits from here and there.

Getting started

First of all, you need to mirror the SVN repository. This takes quite some time, but only needs to be done once. Run this command:

% svk mirror http://svn.secondlife.com/svn/linden //mirror/secondlife
% svk sync //mirror/secondlife

This will make SVK read the whole LL repository, with all its branches and revisions and store them on your hard disk (in your home directory).

When you're done, do the same to a repository you can write to (since you can't write to the LL one):

% svk mirror http://svn.example.com/sl //mirror/myproject
% svk sync //mirror/myproject

//mirror/secondlife and //mirror/myproject are special paths used by SVK. They're links to the actual repositories. Doing something to //mirror/myproject would result in changes being sent to your SVN server.

Creating the first branch

First, copy the branch you want to work with to your own server:

% svk mkdir //mirror/myproject/branches
% svk mkdir //mirror/myproject/branches/buildfixes
% svk smerge -B //mirror/secondlife/release //mirror/myproject/branches/buildfixes

"smerge" stands for "smart merge" and is a very cool thing. It intelligently merges starting from the part where you left. Since there's no common ancestor to both //mirror/secondlife/release and //mirror/myproject/branches/buildfixes (there can't be, since buildfixes is empty), you need to tell it to merge everything with the -B argument. After this initial step, further merges can be performed automatically.

Here we're creating a "buildfixes" branch on our own server, starting from the LL release source. I use the "buildfixes" branch as a starting point for all the others. It contains only the changes needed to get the source to build.

Creating local branches

A very good thing about SVK is that you can do useful work without a network connection. First, we'll create copies of the repositories for local use:

% svk copy //mirror/secondlife //ll/
% svk copy //mirror/myproject //myproject/

Changes made to these aren't automatically propagated to the SVN servers. They're remembered until you want to push them to the server. This means that you can sit on a plane without any sort of network connection, and still use the usual SVN commands -- check the log, commit, revert changes, etc.

Creating a new development branch

Now that the buildfixes branch is created, we can create another starting from it:

% svk mkdir //myproject/trunk
% svk smerge -B //myproject/branches/buildfixes //myproject/trunk

Notice how this time there won't be any network access. To push the changes to your server, do:

% svk smerge //myproject //mirror/myproject

Now suppose you updated the buildfixes branch, and want to propagate the changes. It's simple:

% svk smerge //myproject/branches/buildfixes //myproject/trunk

svk smerge keeps track of what was the last merge, and it automatically merges changes that weren't merged yet. Unlike with SVN, you don't need to keep track of revision numbers of make tags, it automatically does the right thing for you.

Merging new LL code

Now suppose LL releases a new revision and you want to merge it with your changes. This is easy:

% svk sync //mirror/secondlife
% svk smerge //mirror/secondlife //ll
% svk smerge //mirror/secondlife //myproject/branches/buildfixes

Since the buildfixes branch changed, you can propagate the changes further to the trunk branch:

% svk smerge //myproject/branches/buildfixes //myproject/trunk

And push it to your SVN server:

% svk smerge //myproject //mirror/myproject