Difference between revisions of "Autobuild/How To"

Latest revision as of 04:13, 28 April 2015

Autobuild

Autobuild is a framework for maintaining and building projects and managing dependencies among them. This page is a tutorial introduction to basic use of Autobuild. For an overview, see Autobuild.

Using Autobuild to build a project

This section discusses how to configure Autobuild to build a source package. It assumes you have a source distribution and a working build system like make or cmake.

Before you start

Build for your platform(s)

Autobuild is not a build system like make or cmake. It is not intended to solve the nitty-gritty problem of building code on particular platforms; it is designed to provide a common interface for builds and packaging on top of platform-specific tools.

For each library or application you want to manage with Autobuild you need a build (and possibly configure) command for every platform you support. For example, typical Linux packages use makefiles and are built by calling the make command. On Windows you may call devenv.com passing a project file as an argument. In subsequent sections you will learn how to configure Autobuild so it can run your platform configuration and build commands.

Creating a new configuration

The following steps create an autobuild configuration (conventionally autobuild.xml) for your project. That configuration describes the project output, each of the projects dependencies (also referred to as installables), and the platform specific steps required to configure and build the project.

Change directories to the root of your package (or wherever you would like the configuration file to live) and enter this command:

autobuild edit package

This starts an interactive session in which you may fill in the general package details. You must provide a name and the location of a file (relative to the build directory) of a file that will contain the project version number at the completion of the build. If your project is intended to produce an archive package for use as an installable in other projects, you will also want to specify a license and license file. We will ignore the other fields for now. As an alternative to the interactive session, you may add or change package fields all from the command line by passing field=value arguments on the command line. For all edit commands, the interactive mode will be used when no attribute arguments are passed and the non-interactive mode otherwise. For example to create our configuration file non-interactively, use the following command to create the test package version 1.0 using an MIT license:

autobuild edit package name=test license=MIT license_file=LICENSES/test.txt version_file=VERSION.txt

Configuring platform builds

Now that you have defined basic attributes for the package you need to configure the platforms on which to build the library. At this level you can really only configure the build directory because you may define multiple build configurations (see below). The build directory is the directory in which Autobuild will run the build and configure commands and is presumed to be where any build product will be generated. You can, for example, configure the MacOS X platform with the command:

autobuild edit platform name=darwin build_directory=build

Note that for relative paths, the root is always the location of the autobuild configuration file.

Now you need to add a build configuration which includes the build and configure commands which autobuild will invoke to make the package. Assuming you are dealing with a UNIX style package, you typically just call configure and make from the root directory to make the package. You can define the Release configuration to do just that using the commands:

autobuild edit configure platform=darwin name=Release command=../configure
autobuild edit build platform=darwin name=Release command=make options='--directory=..' default=True

You must run these commands in the build directory, so you must the path or add an option as appropriate to work select the root directory. The configure and build commands may take three attributes: command, arguments, and options. These are concatenated to produce the actual command invoked in the shell. The build configuration also supports the default option which flags a build configuration to be build by default if no configuration is specified when the autobuild build command is invoked. You may repeat these steps for other build configurations like Debug with different make and configure options to, for example, build a debug version of the library. You now have enough autobuild configuration to build a simple package for the Darwin platform. Simply run this command to configure and make the package:

autobuild build

Building on multiple platforms

To support other platforms you need only repeat the package, config, and build configuration steps for each platform.

In some cases you may find that different platforms share common command, arguments, or options. For example, a library configured by the cross-platform cmake tool may use the same configuration command on all platforms. In that case you can define the special platform common to define the commonalities. Attributes from build configurations in the common platform are inherited by those for the working platform.

If a command is defined for a platform for a build configuration like Release for both a working platform like darwin that platform will inherit from common. This means that if the arguments or command are not specified in the working platform build configuration for either the build or configure command, then the value from the common will be used. Conversely, if the arguments or command attribute is defined in the working platform, it will supersede what is specified in common. However, the options for the commands from common will be concatenated with those specified in the working platform configuration with the common options preceding the working. Specifying a common platform can allow you to specify commands once per build configuration enabling a single point of truth for cross platform tools.

Adding dependencies

If your package depends on other archives produced by autobuild in other projects, you can instruct autobuild to download and install those archives. First you will need to provide information in the autobuild configuration to identify and locate any package dependencies. You can add a package description of the installable archive using the installables command like the following

autobuild installables add GL url=http://s3.amazonaws.com/viewer-source-downloads/install_pkgs/GL-darwin-20101004.tar.bz2 hash=0b7c1d43dc2b39301fef6c05948fb826

This creates a new entry in the installables dictionary that includes the information needed to download and verify the GL package for the darwin platform. You can add downloads for other platforms using the installables edit command

autobuild installables edit GL platform=windows 
 url=http://s3.amazonaws.com/viewer-source-downloads/install_pkgs/GL-windows-20101001a.tar.bz2 hash=a94538d064cd0a235b2a95389e7e8ee8

Once a package has been added to the installables configuration for the platform you are working on, you can download and install it using the Autobuild install command

autobuild install GL

The package contents will be downloaded and extracted into the packages directory inside the build directory configured for the working platform. Autobuild will conveniently track which packages are installed and what files were extracted only installing or updating files as needed. If you no longer need a package, uninstall it with the command

autobuild uninstall GL

If you update a package by changing its configuration in the installables configuration, Autobuild will automatically clean up the older version so typically you will not need to uninstall packages.

Using Autobuild to construct an archive

Constructing the manifest and packaging

Autobuild has the ability to construct a compressed tar archives with the autobuild package command, but to do that you must tell Autobuild what to include in the archive. Typically you will be packaging an archive of build product generated by the Autobuild build command, but that is not a requirement. Minimally you will need an Autobuild configuration with the name, license, license_file, and copyright attributes set and a platform configuration for the platform for which you wish to build and archive. You may then use the Autobuild manifest command to add files which should be included in the archive. For example, this command adds the pattern 'include/*.h' to the manifest for the platform you are currently running (use the -p option to select another platform).

autobuild manifest add 'include/*.h'

Autobuild supports simple glob patterns, so this pattern instructs autobuild to add all .h header files in the include directory to the archive. The root directory for all paths is assumed to be the build directory configured for the platform. Use only relative paths in your manifest. Additional files may be added by calling manifest add again as needed (note that the add instruction can accept more than one pattern argument). Note that license file will automatically be included in the archive even if it is not explicitly listed in the manifest. If a file is to be included in every archive regardless of platform, you may add it to the manifest of the common platform.

Once you have added all the files required by the archive listed in the manifest, you can generate the archive using the command:

autobuild package

This command will generate an archive in the directory containing the Autobuild configuration file. Unless set using the --archive-name option, the archive will be named using the convention <name>-<version>-<platform>-<buildid>.tar.bz2, e.g. test-1.0-darwin-201010201423.tar.bz2 for the package described in this tutorial.

Laying out build product

Autobuild places no constraints on the layout of build product bundled into an archive, but there are some recommended conventions. The following table shows the recommended directory under which to place files of different types.

File type	Directory
license	LICENSES
header	include
libraries	lib
executables	bin
scripts	scripts

For more details, see Autobuild/Package_Layout. Follow these conventions to help package users easily configure their build systems to find your files. For types not listed here, install files following well-known conventions.

@@ Line 1: / Line 1: @@
-Autobuild is an in-house framework for maintaining and building libraries.  ('''Warning''': Our tool is distinct from [http://josefsson.org/autobuild/ GNU Autobuild] but they are similar enough to cause confusion.)  It acts as director providing a common interface to build and package libraries, but it is not a build system like make or cmake.  You will still need platform specific make, cmake, or project files to actually configure and build your library.  Autobuild will, however, allow you invoke these commands and package the product with a common interface. Linden Lab Autobuild is designed as a replacement for the old [https://svn.lindenlab.com/svn/lindenlib/trunk lindelib] policies, doing the right thing so you don't have to.
+{{Autobuild Nav}}
-== Getting Autobuild ==
+Autobuild is a framework for maintaining and building projects and managing dependencies among them.
-Auotobuild is available as Mercurial repository which you may check out from http://bitbucket.org/lindenlab/autobuild.  You can either run it directly from the checkout or install it as a normal python package using 'setup.py install'.
+This page is a tutorial introduction to basic use of Autobuild.  For an overview, see [[Autobuild]].
-== Using autobuild to build a library ==
+== Using Autobuild to build a project ==
-In this section we discuss how to configure autobuild to build a source package.  We will assume that you have  a source distribution with a build system like make or [http://www.cmake.org/ cmake] up and working.
+This section discusses how to configure Autobuild to build a source package.  It assumes you have a source distribution and a working build system like '''[http://www.gnu.org/software/make/ make]''' or '''[http://www.cmake.org/ cmake]'''.
 === Before you start ===
 ==== Build for your platform(s) ====
-Autobuild is not a build system like [http://www.gnu.org/software/make/ make] or  [http://www.cmake.org/ cmake]. It is not intended to solve the nitty gritty problem of building code on particular platforms; it is designed to provide a common interface for builds and packaging on top of platform specific tools.  You will need a single build (and possibly configure) command that actually complies the library or application you wish to manage with autobuild for every platform you support.  For example, typical linux packages use make files and may be built by simply calling the <nowiki>make</nowiki> command.   On windows you may call ''devenv.exe'' passing a project file as an argument.  In subsequent sections you will learn how to configure autobuild so that it can run your platform configuration and build commands.
+Autobuild is not a build system like '''[http://www.gnu.org/software/make/ make]''' or  '''[http://www.cmake.org/ cmake]'''. It is not intended to solve the nitty-gritty problem of building code on particular platforms; it is designed to provide a common interface for builds and packaging on top of platform-specific tools.
-==== Understand autobuild ====
+For each library or application you want to manage with Autobuild  you need a build (and possibly configure) command for every platform you support.  For example, typical Linux packages use makefiles and are built by calling the <code>make</code> command.   On Windows you may call ''devenv.com'' passing a project file as an argument.  In subsequent sections you will learn how to configure Autobuild so it can run your platform configuration and build commands.
-Before  you start it will be helpful if you familiarize yourself with the autobuild [[Autobuild/Class Model | class model]].  This page describes the structure of an autobuild configuration file explaining the function of the various elements it contains.  The commands that follow in this document will follow the structure of the class model, so they may be easier to understand if you refer to it while going through this  page.
-=== Basic build configuration ===
+=== Creating a new configuration ===
-The following steps will help you configure autobuild to run your package's configure and build commands by invoking nothing more than
+The following steps create an autobuild configuration (conventionally <tt>autobuild.xml</tt>) for your project. That configuration describes the project output, each of the projects dependencies (also referred to as ''installables''), and the platform specific steps required to configure and build the project.
-  autobuild build
+Change directories to the root of your package (or wherever you would like the configuration file to live) and enter this command:
-To start you need to create an autobuild.xml configuration file.  Change directories to the root of your package (or wherever you would like the configuration file to live) and invoke
+ autobuild edit package
-  autobuild edit package
+This starts an interactive session in which you may fill in the general package details.  You must provide a name and the location of a file (relative to the build directory) of a file that will contain the project version number at the completion of the build. If your project is intended to produce an archive package for use as an installable in other projects, you will also want to specify a license and license file.  We will ignore the other fields for now.  As an alternative to the interactive session, you may add or change package fields all from the command line by passing ''field=value'' arguments on the command line.  For all edit commands, the interactive mode will be used when no attribute arguments are passed and the non-interactive mode otherwise.  For example to create our configuration file non-interactively, use the following command to create the '''test''' package version 1.0 using an MIT license:
-This will start an interactive session in which you may fill in the general package details.  Minimally you should provide a name and a version.  If you intend to package your build into an archive, you will also want to specify a license and license file.  We will ignore the other fields for now.  As an alternative to the interactive session, you may add or change package fields all from the command line by passing ''field=value'' arguments on the command line.  For all edit commands, the interactive mode will be used when no attribute arguments are passed and the non-interactive mode otherwise.  For example to create our configuration file non-interactively you can invoke
+ autobuild edit package name=test license=MIT license_file=LICENSES/test.txt version_file=VERSION.txt
-  autobuild edit package name=test license=MIT license_file=LICENSES/test.txt version=1.0
+=== Configuring platform builds ===
+Now that you have defined basic attributes for the package you need to configure the platforms on which to build the library.  At this level you can really only configure the build directory because you may define multiple build configurations (see below).  The build directory is the directory in which Autobuild will run the build and configure commands and is presumed to be where any build product will be generated.  You can, for example, configure the MacOS X platform with the command:
-to create the '''test''' package version 1.0 using an MIT license.  Now that we have defined some of the basic attributes for the package we need to configure the platforms on which we intend for the library to be built.  At this level we can really only configure the build directory because we may define multiple build configurations (see below).  The build directory is the directory in which autobuild will run the build and configure commands and is presumed to be where any build product will be generated.  You can, for example,  configure the MacOS X platform with
+  autobuild edit platform name=darwin build_directory=build
-  autobuild edit platform name=darwin build_directory=build
+Note that for relative paths, the root is always the location of the autobuild configuration file.
-Note that for relative paths, the root is always the location of the autobuild configuration file.  Now you need to add a build configuration which includes the build and configure commands which autobuild will invoke to make the package.   Assuming you are dealing with a UNIX style package, you typically just call ''configure'' and ''make'' from the root directory to make the package.  You can define the '''Release''' configuration to do just that using
+Now you need to add a build configuration which includes the build and configure commands which autobuild will invoke to make the package.   Assuming you are dealing with a UNIX style package, you typically just call ''configure'' and ''make'' from the root directory to make the package.  You can define the '''Release''' configuration to do just that using the commands:
-  autobuild edit configure platform=darwin name=Release command=../configure
+ autobuild edit configure platform=darwin name=Release command=../configure
-  autobuild edit build platform=darwin name=Release command=make options='--directory=..' default=True
+ autobuild edit build platform=darwin name=Release command=make options='--directory=..' default=True
-Note that these commands are run in the build directory so you need to set the path or add an option as appropriate to work select the root directory.  The configure and build commands may take three attributes: '''command''', '''arguments''', and '''options'''.  These are concatenated to produce the actual command invoked in the shell.   The build configuration also supports the '''default''' option which flags a build configuration to be build by default if no configuration is specified when the autobuild ''build'' command is invoked.  You may repeat these steps for other build configurations like '''Debug''' with different ''make'' and ''configure'' options to, for example, build a debug version of the library. You now have enough autobuild configuration to build a simple package for the Darwin platform.  Simply run
+You must run these commands in the build directory, so you must the path or add an option as appropriate to work select the root directory.  The configure and build commands may take three attributes: '''command''', '''arguments''', and '''options'''.  These are concatenated to produce the actual command invoked in the shell.   The build configuration also supports the '''default''' option which flags a build configuration to be build by default if no configuration is specified when the autobuild ''build'' command is invoked.  You may repeat these steps for other build configurations like '''Debug''' with different ''make'' and ''configure'' options to, for example, build a debug version of the library. You now have enough autobuild configuration to build a simple package for the Darwin platform.  Simply run this command to configure and make the package:
-  autobuild build
+ autobuild build
-to configure and make the package.  To support other platform you need only repeat the package, config, and build configuration steps for the platform.
+=== Building on multiple platforms ===
+To support other platforms you need only repeat the package, config, and build configuration steps for each platform.
-In some cases you may find that your different platforms share a common command, arguments, or options.  For example,  a library configured by [http://www.cmake.org/ cmake] may use the same configuration command on all platforms because it is a cross-platform tool.  In that case you can define the special platform '''common''' to define the commonalities.  Attributes from build configurations in the '''common''' platform are inherited by those for the working platform.  If a command is defined for a platform for a build configuration like '''Release''' for both a working platform like '''darwin''' that platform will inherit from '''common'''.  This means that if the ''arguments'' or ''command'' is not specified in the working platform build configuration for either the '''build''' or '''configure''' command then value from the '''common''' will be  used.  Conversely, if the ''arguments'' or ''command'' attribute is defined in the working platform, it will supersede what is specified in '''common'''.  However, the ''options'' for the commands from '''common''' will be concatenated with those specified in the working platform configuration with  the common options preceding the working.   Specifying a '''common''' platform can allow you to specify commands once per build configuration enabling a single point of truth for cross platform tools.
+In some cases you may find that different platforms share common command, arguments, or options.  For example, a library configured by the cross-platform [http://www.cmake.org/ cmake] tool may use the same configuration command on all platforms.  In that case you can define the special platform '''common''' to define the commonalities.  Attributes from build configurations in the '''common''' platform are inherited by those for the working platform.
+If a command is defined for a platform for a build configuration like '''Release''' for both a working platform like '''darwin''' that platform will inherit from '''common'''.  This means that if the ''arguments'' or ''command'' are not specified in the working platform build configuration for either the '''build''' or '''configure''' command, then the value from the '''common''' will be  used.  Conversely, if the ''arguments'' or ''command'' attribute is defined in the working platform, it will supersede what is specified in '''common'''.  However, the ''options'' for the commands from '''common''' will be concatenated with those specified in the working platform configuration with  the common options preceding the working.   Specifying a '''common''' platform can allow you to specify commands once per build configuration enabling a single point of truth for cross platform tools.
 === Adding dependencies ===
-If your package depends on other archives, you can instuct autobuild to download and install those archives.  This feature is useful for binary archives bundled as compressed tar files dowloadable from a url (typically generated by autobuild).   First you will need to provide information in the autobuild configuration to identify and locate any package dependencies.  You can add a package description of the installable archive using the ''installables'' command like the following
+If your package depends on other archives produced by autobuild in other projects, you can instruct autobuild to download and install those archives. First you will need to provide information in the autobuild configuration to identify and locate any package dependencies.  You can add a package description of the installable archive using the ''installables'' command like the following
-   autobuild installables add GL license=GPL license_file=LICENSES/GL.txt platform=darwin <br> url=http://s3.amazonaws.com/viewer-source-downloads/install_pkgs/GL-darwin-20101004.tar.bz2 hash=0b7c1d43dc2b39301fef6c05948fb826
+ autobuild installables add GL url=http://s3.amazonaws.com/viewer-source-downloads/install_pkgs/GL-darwin-20101004.tar.bz2 hash=0b7c1d43dc2b39301fef6c05948fb826
 This creates a new entry in the installables dictionary that includes the information needed to download and verify the '''GL''' package for the '''darwin''' platform.  You can add downloads for other platforms using the installables edit command
-  autobuild installables edit GL platform=windows <br> url=http://s3.amazonaws.com/viewer-source-downloads/install_pkgs/GL-windows-20101001a.tar.bz2 hash=a94538d064cd0a235b2a95389e7e8ee8
+ autobuild installables edit GL platform=windows <br> url=http://s3.amazonaws.com/viewer-source-downloads/install_pkgs/GL-windows-20101001a.tar.bz2 hash=a94538d064cd0a235b2a95389e7e8ee8
-Once an package has been added to the installables configuration for the platform you are working on, you can download it using the autobuild ''install'' command
+Once a package has been added to the installables configuration for the platform you are working on, you can download and install it using the Autobuild ''install'' command
-  autobuild install GL
+ autobuild install GL
-The package contents will be downloaded and extracted into the '''packages''' directory inside the build directory configured for the working platform.  Autobuild will conveniently track which packages are installed and what files were extracted only installing or updating files as needed.  A package may be uninstalled with
+The package contents will be downloaded and extracted into the '''packages''' directory inside the build directory configured for the working platform.  Autobuild will conveniently track which packages are installed and what files were extracted only installing or updating files as needed.  If you no longer need a package, uninstall it with the command
   autobuild uninstall GL
-if it is no longer needed.  If you update a package by changing its configuration in the installables configuration, autobuild will automatically clean up the older version so typically you will not need to uninstall packages.
+If you update a package by changing its configuration in the installables configuration, Autobuild will automatically clean up the older version so typically you will not need to uninstall packages.
-== Using autobuild to construct an archive ==
+== Using Autobuild to construct an archive ==
 === Constructing the manifest and packaging ===
-Autobuild has the ability to construct a compressed tar archives with the autobuild ''package'' command, but to do that you must tell autobuild what to include in the archive.  Typically you will be packaging an archive of build product generated by the autobuild ''build'' command, but that is not a requirement.  Minimally you will need an autobuild configuration with the '''name''', '''license''', '''license_file''', and '''version''' attributes set and a platform configuration for the platform for which you wish to build and archive.  You may then use the autobuild ''manifest'' command to add files which should be included in the archive.  For example
+Autobuild has the ability to construct a compressed tar archives with the autobuild ''package'' command, but to do that you must tell Autobuild what to include in the archive.  Typically you will be packaging an archive of build product generated by the Autobuild ''build'' command, but that is not a requirement.  Minimally you will need an Autobuild configuration with the '''name''', '''license''', '''license_file''', and '''copyright''' attributes set and a platform configuration for the platform for which you wish to build and archive.  You may then use the Autobuild ''manifest'' command to add files which should be included in the archive.  For example, this command adds the pattern 'include/*.h' to the manifest for the platform you are currently running (use the ''-p'' option to select another platform).
-  autobuild manifest add include/*.h
+  autobuild manifest add 'include/*.h'
-will add the pattern 'include/*.h' to the manifest for the platform you are currently running (use the ''-p'' option to select another platform).  Autoubuild supports simple glob patterns, so this pattern instructs autobuild to add all .h header files in the include directory to the archive.  The root directory for all paths is assumed to be the build directory configured for the platform.  You are strongly encouraged to use only relative paths in your manifest.  Additional files may be added by calling ''manifest add'' again as needed (note that the ''add'' instruction can accept more than one pattern argument).  Note that license file will automatically be included in the archive even if it is not explicitly listed in the manifest.  If a file is to be included in every archive regardless of platform, you may add it to the manifest of the '''common''' platform.
+Autobuild supports simple glob patterns, so this pattern instructs autobuild to add all .h header files in the include directory to the archive.  The root directory for all paths is assumed to be the build directory configured for the platform.  ''Use only relative paths in your manifest.'' Additional files may be added by calling ''manifest add'' again as needed (note that the ''add'' instruction can accept more than one pattern argument).  Note that license file will automatically be included in the archive even if it is not explicitly listed in the manifest.  If a file is to be included in every archive regardless of platform, you may add it to the manifest of the '''common''' platform.
-Once you have added all the files required by the archive listed in the manifest, you can generate the archive using
+Once you have added all the files required by the archive listed in the manifest, you can generate the archive using the command:
-  autobuild package
+ autobuild package
-This command will generate an archive in the directory where the autobuild configuration file resides.  Unless set using the ''--archive-name'' option, the archive will be named using the convention <nowiki><name>-<version>-<platform>-<date>.tar.bz2</nowiki>, e.g. test-1.0-darwin-20101020.tar.bz2 for the package described in this tutorial.
+This command will generate an archive in the directory containing the Autobuild configuration file.  Unless set using the ''--archive-name'' option, the archive will be named using the convention <nowiki><name>-<version>-<platform>-<buildid>.tar.bz2</nowiki>, e.g. test-1.0-darwin-201010201423.tar.bz2 for the package described in this tutorial.
 === Laying out build product ===
@@ Line 86: / Line 89: @@
 Autobuild places no constraints on the layout of build product bundled into an archive, but there are some recommended conventions.  The following table shows the recommended directory under which to place files of different types.
-{|border="1" style="background:white"
+{|border="1" class=lltable
-|File type
+|--
-|Directory
+!File type
+!Directory
 |-
 |license
-||LICENSES
+|LICENSES
 |-
 |header
-||include
+|include
 |-
 |libraries
-||lib
+|lib
 |-
 |executables
-||bin
+|bin
 |-
 |scripts
-||scripts
+|scripts
 |}
-For  more details, see [[Autobuild/Package_Layout]]. By following these conventions you will help package users easily configure their build systems to find your files. For types not listed here it is recommended that you install files following well known conventions.
+For  more details, see [[Autobuild/Package_Layout]]. Follow these conventions to help package users easily configure their build systems to find your files. For types not listed here, install files following well-known conventions.
-[[Category:Open Source Portal]]
+[[Category:Autobuild]] [[Category:Open Source Portal]]