Autobuild 1.0

This page documents the changes between autobuild version 0.8 (the current production version), and 0.9.

There are few things that need to be done to the viewer build process (and possibly the simulator build process) before this version can become the default version.

The intent is that it is as much as possible backwards compatible:

archives produced by 0.9 can be used by 0.8 (there is some extra content that the older version ignores),
archives produced by 0.8 can be used by 0.9, but will be considered 'dirty' (see below)

Goals

Improve integrity of builds by doing additional dependency checks
Improve ease of use for developers by reducing what a developer must do to change a dependency/installable used in a build
Provide additional subcommands and options to inform the developer regarding dependencies, both direct and indirect

Removed "Features"

--as-source: This option to the install subcommand seems to have been intended to mimic sub-repos by checking out package sources as a subdirectory in the current source tree. As far as I've been able to determine, it has never been used, and probably did not work correctly. This entire notion has been removed (reducing some code complexity in the process).
--skip-license-check: The package description must now include a license value; the --skip-license-check option is now a no-op (except that it prints a warning that it no longer works). Any existing archives that lack a license will have to be rebuilt.
upload subcommand: This was useful only to Lindens, and as far as I can tell not used by them. It was redundant with features of the build farm that handle archives, and has been removed.

New Features

Archive Metadata

The top level of any package archive file (.tar.bz2 or .zip) contains a package metadata file generated by the 'build' subcommand and included in the archive by the 'package' subcommand (it is not listed in the package manifest):

autobuild-package.xml

It is, like the autobuild.xml configuration file, an xml-encoded llsd map with the top level attributes below; the values are generated (shown in parenthesis) or taken from the named attribute in autobuild configuration file used to create the package:

Metadata Contents

package_description: map of package attributes from the autobuild configuration used to build the archive
build_id: unique integer provided by --id; defaults to a date/time value
configuration: provided by --configuration (-c)
platform: autobuild platform identifier
manifest: array of the path names installed by the package; since the autobuild-package.xml file is not installed, it is not included in the manifest
dependencies: map of metadata for packages this depends on, keyed by dependency name
dirty: (optional) bool: true indicates that this package was built using a 'dirty' dependency, or from components not imported as a downloaded autobuild package (such a locally built package)
type: configuration file type ('metadata')
version: version of the metadata contents: 1

Easier Installable Changes

Because each archive now includes the metadata needed for configuration, you no longer need to specify all of its properties when using the installable add or installable edit subcommands; you can just specify the url of the archive:

autobuild installable add https://example.com/path/somelib-1.2.1-darwin-12345.tar.bz2

autobuild will fetch and examine the archive to determine everything it needs to know in order to add it to the configuration (if you want to specify the hash as a check, you may do so; if you don't, it will be computed from the downloaded archive.

Install Checks

When the 'autobuild install' command is used to add an installable, the requisite installable archive is fetched, its hash checked against that in the configuration file, and before any other files are unpacked from it, the package metadata is extracted. The absence of metadata causes the installed archive to be considered 'dirty': by default, a warning message is logged and the resulting build is also marked 'dirty'; if the --clean-only option was specified, this is a fatal error.

The initial checks against the metadata confirm that the package contents match what is configured (in all of the following, "METADATA/" refers to the value from the installable package metadata, "CONFIG/" refers to the value from the current autobuild configuration):

METADATA/name == CONFIG/installable:<package>:name
METADATA/platform == (current platform, or "common")

For each of the entries in the METADATA/manifest map, ensure that the file does not already exist (this indicates that two different packages are installing the same file). In version 0.8, a package could overwrite files installed by some other package; this is now a fatal error.

Check the transitive dependencies (ensure that all packages included in the current build are using the same version of any installable): failure is fatal.

For each of the entries in the METADATA/installables map:

<package>/build_id == CONFIG/installable:<package>:build_id
<package>/url  	== CONFIG/installable:<package>:url
<package>/hash 	== CONFIG/installable:<package>:hash

If the tests above pass, the package is installed and all of the metadata from the installed package, including its dependencies, is added to that for the current build.

"Graph" Subcommand

The new subcommand graph displays a graph of the transitive dependencies for a package. Packages are drawn as dashed ovals if they are 'dirty' and solid squares if not. This is especially useful if you need to change a package that is used by more than one dependency to work out what other packages you will need to rebuild.

Build Ids

The build and configure subcommands now take a --id argument to pass a build identifier string. This identifier is used in constructing the archive name; if not specified, it defaults to a date/time value. The id may be specified using the AUTOBUILD_ID environment variable; in any event, that variable is set for all processes run by autobuild so that it is available to them. Builds in the Linden build farm will set this to the codeticket number.

Clean Only

The build and configure subcommands now take a --clean-only option to require that the build not include any 'dirty' packages. This may also be specified by setting the AUTOBUILD_CLEAN_ONLY to 'true' (which is set for any processes under autobuild when required).

Someday we will be able to do all builds this way...

Install Options

The install subcommand has some new options for displaying information about installed packages:

--list-install-urls: Lists the archive urls for all installed packages
--list-dirty: Lists the names of 'dirty' installed packages
--what-installed file-path: Lists the archive responsible for installing the specified file; useful when finding conflicts between packages

It also has an option for making installation of a locally built package easier (handy during development):

--local archive: Installs the specified archive file in place of the version in the autobuild configuration. This marks the resulting build as 'dirty' but is useful during development.

Using Version 0.9

You can check out this version of autobuild from:

https://bitbucket.org/oz_linden/autobuild-metadata

Install and use as you would the previous version.

Contact Oz for how to use it on TeamCity.