This page documents the changes between autobuild version 0.8 (the current production version), and 1.0. We are converting to using 1.0 as a part of the tools upgrade project, which is also upgrading to building with Cmake 3.1, Xcode 6.1 (clang), and Visual Studio 2013.

Goals

Improve integrity of builds by doing additional dependency checks
Improve ease of use for developers by reducing what a developer must do to change a dependency/installable used in a build
Provide additional subcommands and options to inform the developer regarding dependencies, both direct and indirect
Tighten requirements for legal attributes like copyright and licenses, and provide easier ways to use that data in a subsequent build

The intent is that it is as much as possible backwards compatible:

archives produced by 1.0 can be used by 0.8 (there is some extra content that the older version ignores),
archives produced by 0.8 can be used by 1.0, but will be considered 'dirty' (see below)

Some attributes of the package description that were previously optional are now required, and there is one new attribute that will have to be added (see #Version Number Handling)

Using Version 1.0

You can check out this version of autobuild from:

https://bitbucket.org/lindenlab/autobuild-1.0

You should use Python 2.7 for autobuild 1.0 (version 2.6.6 may work for most subcommands, but will complain).

To install with the appropriate python package dependencies, use:

pip install 'hg+https://bitbucket.org/lindenlab/autobuild-1.0#egg=autobuild'

depending on how your permissions are defined and whether or not you are working within a python virtualenv, you may need to run that under the sudo command.

Changes

Removed "Features"

--as-source: This option to the install subcommand seems to have been intended to mimic sub-repos by checking out package sources as a subdirectory in the current source tree. As far as I've been able to determine, it has never been used, and probably did not work correctly. This entire notion has been removed (reducing some code complexity in the process).
--skip-license-check: The package description must now include a license value; the --skip-license-check option is now a no-op (except that it prints a warning that it no longer works). Any existing archives that lack a license will have to be rebuilt.
upload subcommand: This was useful only to Lindens, and as far as I can tell not used by them. It was redundant with features of the build farm that handle archives, and has been removed.

New Features

Archive Metadata

The top level of any package archive file (.tar.bz2 or .zip) contains a package metadata file generated by the 'build' subcommand and included in the archive by the 'package' subcommand (it is not listed in the package manifest):

autobuild-package.xml

It is, like the autobuild.xml configuration file, an xml-encoded llsd map with the top level attributes below; the values are generated (shown in parenthesis) or taken from the named attribute in autobuild configuration file used to create the package:

Metadata Contents

package_description: map of package attributes from the autobuild configuration used to build the archive
build_id: unique integer provided by --id; defaults to a date/time value
configuration: provided by --configuration (-c)
platform: autobuild platform identifier
manifest: array of the path names installed by the package; since the autobuild-package.xml file is not installed, it is not included in the manifest
dependencies: map of metadata for packages this depends on, keyed by dependency name
dirty: (optional) bool: true indicates that this package was built using a 'dirty' dependency, or from components not imported as a downloaded autobuild package (such a locally built package)
type: configuration file type ('metadata')
version: version of the metadata contents: 1

Easier Installable Changes

Because each archive now includes the metadata needed for configuration, you no longer need to specify all of its properties when using the installable add or installable edit subcommands; you can just specify the url of the archive:

autobuild installable add https://example.com/path/somelib-1.2.1-darwin-12345.tar.bz2

autobuild will fetch and examine the archive to determine everything it needs to know in order to add it to the configuration. If you want to specify the hash as a check, you may do so; if you don't, it will be computed from the downloaded archive.

The 'copyright', 'license', and 'version' elements of an installable description are no longer required when importing new style packages, since they are read from the package metadata. If you do specify them, you'll get messages that they are no longer needed, but then they must match what's in the package when it is installed.

Version Number Handling

Required Update to autobuild.xml The package_description in the autobuild configuration file is now required to specify a 'version_file' attribute; this replaces the 'version' attribute.

Any 'version' attribution in the package_description is now ignored (with a warning).

The value of 'version_file' is the path relative to the build directory of a file containing the package version number value. The build sub-command reads the version_file after the build command has run and puts its contents into the metadata/package_description/version attribute. The goal is to have the build script extract the version from the sources rather than duplicating it in the autobuild configuration file (with the attendant risk of it being out of sync).

The autobuild config file and the build script for each package will have to be modified to follow this new convention.

Install Checks

When the install subcommand is used to add an installable, the requisite installable archive is fetched, its hash checked against that in the configuration file, and before any other files are unpacked from it, the package metadata is extracted. The absence of metadata causes the installed archive to be considered 'dirty': by default, a warning message is logged and the resulting build is also marked 'dirty'; if the --clean-only option was specified, this is a fatal error.

The initial checks against the metadata confirm that the package contents match what is configured (in all of the following, "METADATA/" refers to the value from the installable package metadata, "CONFIG/" refers to the value from the current autobuild configuration):

METADATA/name == CONFIG/installable:<package>:name
METADATA/platform == (current platform, or "common")

For each of the entries in the METADATA/manifest map, ensure that the file does not already exist (this indicates that two different packages are installing the same file). In version 0.8, a package could overwrite files installed by some other package; this is now a fatal error.

Check the transitive dependencies (ensure that all packages included in the current build are using the same version of any installable): failure is fatal.

For each of the entries in the METADATA/installables map:

<package>/build_id == CONFIG/installable:<package>:build_id
<package>/url  	== CONFIG/installable:<package>:url
<package>/hash 	== CONFIG/installable:<package>:hash

If the tests above pass, the package is installed and all of the metadata from the installed package, including its dependencies, is added to that for the current build.

Installing a package now requires that the package metadata include the following elements in package_description:

version
name
copyright
license
license_file

if any of those are missing, warnings will be printed and the installation will be marked as 'dirty'.

"Graph" Subcommand

The new subcommand graph displays a graph of the transitive dependencies for a package. Packages are drawn as dashed ovals if they are 'dirty' and solid squares if not. This is especially useful if you need to change a package that is used by more than one dependency to work out what other packages you will need to rebuild.

Build Ids

The build and configure subcommands now take a --id argument to pass a build identifier string. This identifier is used in constructing the archive name; if not specified, it defaults to a date/time value. The id may be specified using the AUTOBUILD_ID environment variable; in any event, that variable is set for all processes run by autobuild so that it is available to them. Builds in the Linden build farm will set this to the codeticket number.

'common' Platform Builds

You may now directly specify 'common' as the platform for the build and configure subcommands (either using --platform or through the AUTOBUILD_PLATFORM_OVERRIDE environment variable). This is useful when building source-only packages that are platform independent.

Clean Only

The build and configure subcommands now take a --clean-only option to require that the build not include any 'dirty' packages. This may also be specified by setting the AUTOBUILD_CLEAN_ONLY to 'true' (which is set for any processes under autobuild when required).

Someday we will be able to do all builds this way...

Install Options

The install subcommand has some new options for displaying information about installed packages:

--list-install-urls: Lists the archive urls for all installed packages
--list-dirty: Lists the names of 'dirty' installed packages
--what-installed file-path: Lists the archive responsible for installing the specified file; useful when finding conflicts between packages
--versions: Prints each package name and its version value
--copyrights: Prints each package name and its copyright value

It also has an option for making installation of a locally built package easier (handy during development):

--local archive: Installs the specified archive file in place of the version in the autobuild configuration. This marks the resulting build as 'dirty' but is useful during development. Using this option is equivalent to listing a package name to be installed (so not including the name does not mean that all packages are installed).

Package Command

The package subcommand now accepts a --results-file path option that specifies the name of a file to contain shell style variable assignments for the results for easier and more robust integration with build scripts:

autobuild_package_name="zlib"
autobuild_package_filename="/user/me/work/3p-zlib/zlib-1.2.8.201409080943-darwin-201409080943.tar.bz2"
autobuild_package_md5="e52c11448cf5c12eee86a1d9cabf888e"
autobuild_package_clean=true

It is now a fatal error if the configuration file manifest specifies a file or glob that is not found. In autobuild 0.8 this was not true, so some existing configuration files will need to be fixed to build with 1.0

Autobuild 1.0

Contents