Difference between revisions of "Autobuild/Quick Start"

From Second Life Wiki
Jump to: navigation, search
(What haven't we done)
 
(24 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 +
{{Autobuild Nav}}
 +
==Introduction ==
 
When creating a new autobuild package of a library, it can be unclear where to start.  [[Autobuild How To]] can be a little intimidating.  We're packaging a simple third party library just like all the rest of ours, so we don't need to use autobuild in it's full generality.  We can skip some steps and share script fragments with other library packages.
 
When creating a new autobuild package of a library, it can be unclear where to start.  [[Autobuild How To]] can be a little intimidating.  We're packaging a simple third party library just like all the rest of ours, so we don't need to use autobuild in it's full generality.  We can skip some steps and share script fragments with other library packages.
  
Autobuild is largely self documenting.  You can access help via the commands <tt>autobuild --help</tt> or <tt>autobuild &lt;subcommand&gt; --help</tt>, for example <tt>autobuild edit --help</tt>.  If the help text is unclear, please file a bug.
+
Autobuild is largely self documenting.  You can access help via the commands <code>autobuild --help</code> or <code>autobuild &lt;subcommand&gt; --help</code>, for example <code>autobuild edit --help</code>.  If the help text is unclear, please file a bug.
 
+
Here's a crash course:
+
  
 
==What will we be doing==
 
==What will we be doing==
  
Your packaging efforts will mostly consist of creating or modifying 3 files: build.sh, autobuild.xml, and build-cmd.sh
+
Your packaging efforts will mostly consist of creating or modifying these two files: :
  
;build.sh : Teamcity's (or another unattended build system's) interface to your package, it lists the sequence of autobuild commands to run.
+
;autobuild.xml : Autobuild package definition metadata.  This specifies the actual shell commands that autobuild should execute for each step.  Modify this file using the <code>autobuild edit</code> command.  Editing it by hand can be fragile.
;autobuild.xml : Autobuild package definition metadata.  For now, what you should focus on "This specifies the actual shell commands that autobuild should execute for each step." You should modify this file using the '<tt>autobuild edit</tt>' command.  Editing it by hand will lead to problems.
+
;build-cmd.sh : Shell script you are going to specify as the build command in autobuild.xml.  This script encapsulates a lot of the complexity for third party libraries, so you can just clone it from an existing library and customize it for your package.
;build-cmd.sh : the shell script that we are going to specify as the build command in autobuild.xml
+
  
 
==What do we do==
 
==What do we do==
Get autobuild:
+
[[Autobuild#Getting_Autobuild|Get autobuild]]:
  hg clone http://hg.lindenlab.com/brad/autobuild-trunk
+
  pip install autobuild
cd autobuild-trunk/bin
+
export PATH="$PATH:$(pwd)"
+
cd ../..
+
  
 
Copy the skeleton from an existing package.
 
Copy the skeleton from an existing package.
 
  hg init hello
 
  hg init hello
 
  cd hello
 
  cd hello
  curl -OL http://hg.lindenlab.com/brad/curl-autobuild/raw/tip/BuildParams
+
  curl -OL http://bitbucket.org/lindenlab/3p-zlib/raw/tip/BuildParams
  curl -OL http://hg.lindenlab.com/brad/curl-autobuild/raw/tip/build.sh
+
  curl -OL http://bitbucket.org/lindenlab/3p-zlib/raw/tip/build-cmd.sh
curl -OL http://hg.lindenlab.com/brad/curl-autobuild/raw/tip/build-cmd.sh
+
  chmod +x build-cmd.sh
  chmod +x build.sh build-cmd.sh
+
  curl -OL http://bitbucket.org/lindenlab/3p-zlib/raw/tip/.hgignore
  curl -OL http://hg.lindenlab.com/brad/curl-autobuild/raw/tip/.hgignore
+
  hg add BuildParams build-cmd.sh .hgignore
  hg add BuildParams build.sh build-cmd.sh .hgignore
+
  
Now you'll need to edit build-cmd.sh and customize it for your package in our case, [http://www.gnu.org/software/hello/ GNU Hello].  Mostly this is just a search and replace, although you'll probably need to uncomment the fetch_archive line and the variables it uses (CURL_URL and CURL_MD5).
+
Now you'll need to edit build-cmd.sh and customize it for your package in our case, [http://www.gnu.org/software/hello/ GNU Hello].  Mostly this is just a search and replace (replace occurrences of zlib and ZLIB with the name or your package).  For example, the sed search & replace commands would be <code>s/zlib/hello/g</code> and <code>s/ZLIB/HELLO/g</code> in this case.  You'll likely also need to update the version number variables to match your package's version.
  
Additionally you'll need to update the 3 sections under the case statement with any platform specific quirks for how to build your package natively.  Darwin and Linux can be pretty standardized, but under windows there are a lot of different ways to build your package.  Most of these have been worked through already for one package or another, so talk to a team [[Chopper]] member or scan [[Autobuild/Package Index]] if you need examples to imitate.
+
Additionally you'll need to update the 3 sections under the case statement with any platform specific quirks for how to build your package natively.  Darwin and Linux can be pretty standardized, but under windows there are a lot of different ways to build your package.  Most of these have been worked through already for one package or another. You can use these [[Autobuild Package Examples|Autobuild Examples]] as a starting point.
  
  
In the end, you should come up with something like [[Autobuild/Quick_Start/build-cmd.sh|this]].
+
In the end, you should come up with something like this [[Autobuild/Quick_Start/build-cmd.sh|example build-cmd.sh]]. See [[Build Script Anatomy]].
 +
 
 +
You'll need some information about your package from the upstream package distributor for filling in some of the prompts.  For example you should copy the copyright string and license info verbatim from the upstream distributor.  Additionally we should always write our build script to copy the text of the license into <code>LICENSES/&lt;package name&gt;.txt</code> and specify that location as the "Path to license file"
  
 
Next you'll want to configure autobuild to use your build-cmd.sh.
 
Next you'll want to configure autobuild to use your build-cmd.sh.
 
  autobuild edit package
 
  autobuild edit package
 
  # Name of package> hello
 
  # Name of package> hello
  # Package description> The GNU Hello program produces a familiar, friendly greeting. Yes, this is another implementation of the classic program that prints “Hello, world!” when you run it.
+
  # Package description> The GNU Hello program produces a familiar, friendly greeting. \
 +
Yes, this is another implementation of the classic program that prints “Hello, world!” when you run it.
 
  # Copyright string (as appropriate for your package)> Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
 
  # Copyright string (as appropriate for your package)> Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
 
  # Type of license (as appropriate for your package)> gpl3
 
  # Type of license (as appropriate for your package)> gpl3
Line 49: Line 47:
 
  # Name of config> default
 
  # Name of config> default
 
  # Platform of config> common
 
  # Platform of config> common
  # Command to execute> ../build-cmd.sh
+
  # Command to execute> bash
# Options for command>
+
# Options for command> -c,../build-cmd.sh
 
  # Arguments for command>
 
  # Arguments for command>
 +
# Filter command output>
 +
# Run by default> True
 
  autobuild edit platform name=linux build_directory=stage
 
  autobuild edit platform name=linux build_directory=stage
 
  autobuild edit platform name=darwin build_directory=stage
 
  autobuild edit platform name=darwin build_directory=stage
Line 59: Line 59:
 
  autobuild edit build name=default platform=windows default=true
 
  autobuild edit build name=default platform=windows default=true
  
Now if you run <tt>autobuild print</tt> its output should look like [[Autobuild/Quick_Start/Build Configured|this]]
+
Now if you run <code>autobuild print</code> its output should look like this [[Autobuild/Quick_Start/Build Configured|autobuild print example]]
  
 
Next you'll want to configure the manifest for packaging.
 
Next you'll want to configure the manifest for packaging.
Line 75: Line 75:
 
  autobuild manifest --platform windows add "bin/*.exe"
 
  autobuild manifest --platform windows add "bin/*.exe"
  
Now if you run <tt>autobuild print</tt> its output should look like [[Autobuild/Quick_Start/Manifest Configured|this]]
+
Now if you run <code>autobuild print</code> its output should look like this [[Autobuild/Quick_Start/Manifest Configured|autobuild manifest example]]
  
Now things should be complete (for this simplified example), and you should be able to run <tt>autobuild build && autobuild package</tt> to get a package generated.  Hooray!
+
Now things should be complete (for this simplified example), and you should be able to run <code>autobuild build && autobuild package</code> to get a package generated.  Hooray!
  
 
  hg commit
 
  hg commit
Line 88: Line 88:
 
==What haven't we done==
 
==What haven't we done==
  
# We haven't addressed how to handle dependencies.  For example libcurl depends upon zlib and openssl already having their autobuild packages availible for use.  If you're packaging a new library that has dependencies you should begin at the leaves of the dependency tree.
+
# We haven't addressed how to handle dependencies.  For example libcurl depends upon zlib and openssl already having their autobuild packages available for use.  If you're packaging a new library that has dependencies you should begin at the leaves of the dependency tree.
 +
#:[[Autobuild_How_To#Adding_dependencies]] covers this excellently.
 
# We skipped features of autobuild that aren't useful for the way we're currently packaging third party library packages.
 
# We skipped features of autobuild that aren't useful for the way we're currently packaging third party library packages.
 
## like multiple configurations (Debug/RelWithDebInfo/Release)
 
## like multiple configurations (Debug/RelWithDebInfo/Release)
Line 96: Line 97:
 
# We haven't covered specifically what the ABI requirements are for packages intended to be linked into the viewer.
 
# We haven't covered specifically what the ABI requirements are for packages intended to be linked into the viewer.
  
[[Category:Open Source Portal]]
+
==Simple Example==
 +
 
 +
It is also useful to look at the tiniest possible example of autobuild without the added complexity of building a large project like the Second Life Viewer, which has additional interacting build scripts, CMake and other build time elements that can obscure the picture of what autobuild is doing.  To that end, there is a toy autobuild project here: [https://bitbucket.org/lindenlab/hello_autobuild/src Hello Autobuild] which can be checked out and played with to get a feel for autobuild.  Please do not check back changes to this repo so that everyone gets the same baseline to work from.
 +
 
 +
 
 +
[[Category:Autobuild]] [[Category:Open Source Portal]]

Latest revision as of 10:00, 12 April 2017


Introduction

When creating a new autobuild package of a library, it can be unclear where to start. Autobuild How To can be a little intimidating. We're packaging a simple third party library just like all the rest of ours, so we don't need to use autobuild in it's full generality. We can skip some steps and share script fragments with other library packages.

Autobuild is largely self documenting. You can access help via the commands autobuild --help or autobuild <subcommand> --help, for example autobuild edit --help. If the help text is unclear, please file a bug.

What will we be doing

Your packaging efforts will mostly consist of creating or modifying these two files: :

autobuild.xml 
Autobuild package definition metadata. This specifies the actual shell commands that autobuild should execute for each step. Modify this file using the autobuild edit command. Editing it by hand can be fragile.
build-cmd.sh 
Shell script you are going to specify as the build command in autobuild.xml. This script encapsulates a lot of the complexity for third party libraries, so you can just clone it from an existing library and customize it for your package.

What do we do

Get autobuild:

pip install autobuild

Copy the skeleton from an existing package.

hg init hello
cd hello
curl -OL http://bitbucket.org/lindenlab/3p-zlib/raw/tip/BuildParams
curl -OL http://bitbucket.org/lindenlab/3p-zlib/raw/tip/build-cmd.sh
chmod +x build-cmd.sh
curl -OL http://bitbucket.org/lindenlab/3p-zlib/raw/tip/.hgignore
hg add BuildParams build-cmd.sh .hgignore

Now you'll need to edit build-cmd.sh and customize it for your package in our case, GNU Hello. Mostly this is just a search and replace (replace occurrences of zlib and ZLIB with the name or your package). For example, the sed search & replace commands would be s/zlib/hello/g and s/ZLIB/HELLO/g in this case. You'll likely also need to update the version number variables to match your package's version.

Additionally you'll need to update the 3 sections under the case statement with any platform specific quirks for how to build your package natively. Darwin and Linux can be pretty standardized, but under windows there are a lot of different ways to build your package. Most of these have been worked through already for one package or another. You can use these Autobuild Examples as a starting point.


In the end, you should come up with something like this example build-cmd.sh. See Build Script Anatomy.

You'll need some information about your package from the upstream package distributor for filling in some of the prompts. For example you should copy the copyright string and license info verbatim from the upstream distributor. Additionally we should always write our build script to copy the text of the license into LICENSES/<package name>.txt and specify that location as the "Path to license file"

Next you'll want to configure autobuild to use your build-cmd.sh.

autobuild edit package
# Name of package> hello
# Package description> The GNU Hello program produces a familiar, friendly greeting. \
Yes, this is another implementation of the classic program that prints “Hello, world!” when you run it.
# Copyright string (as appropriate for your package)> Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
# Type of license (as appropriate for your package)> gpl3
# Path to license file relative to package root, if known> LICENSES/hello.txt
# Version> 2.6
hg add autobuild.xml
autobuild edit build
# Name of config> default
# Platform of config> common
# Command to execute> bash
# Options for command> -c,../build-cmd.sh
# Arguments for command>
# Filter command output>
# Run by default> True
autobuild edit platform name=linux build_directory=stage
autobuild edit platform name=darwin build_directory=stage
autobuild edit platform name=windows build_directory=stage
autobuild edit build name=default platform=linux default=true
autobuild edit build name=default platform=darwin default=true
autobuild edit build name=default platform=windows default=true

Now if you run autobuild print its output should look like this autobuild print example

Next you'll want to configure the manifest for packaging.

autobuild manifest --platform common add "include/hello/*.h"
autobuild manifest --platform common add "LICENSES/hello.txt"
autobuild manifest --platform linux add "lib/*.a"
autobuild manifest --platform darwin add "lib/*.a"
autobuild manifest --platform windows add "lib/debug/*.pdb"
autobuild manifest --platform windows add "lib/debug/*.lib"
autobuild manifest --platform windows add "lib/release/*.pdb"
autobuild manifest --platform windows add "lib/release/*.lib"
autobuild manifest --platform linux add "bin/*"
autobuild manifest --platform darwin add "bin/*"
autobuild manifest --platform windows add "bin/*.exe"

Now if you run autobuild print its output should look like this autobuild manifest example

Now things should be complete (for this simplified example), and you should be able to run autobuild build && autobuild package to get a package generated. Hooray!

hg commit

What did we do

  1. We defined a build script that drives the native build of the package in a way that is compatible with our viewer builds and sandboxed from the rest of the system.
  2. We configured autobuild to use that build script on all 3 platforms.
  3. We packaged the result in a tarball consistent with Autobuild/Package_Layout

What haven't we done

  1. We haven't addressed how to handle dependencies. For example libcurl depends upon zlib and openssl already having their autobuild packages available for use. If you're packaging a new library that has dependencies you should begin at the leaves of the dependency tree.
    Autobuild_How_To#Adding_dependencies covers this excellently.
  2. We skipped features of autobuild that aren't useful for the way we're currently packaging third party library packages.
    1. like multiple configurations (Debug/RelWithDebInfo/Release)
    2. like options and arguments to the build command.
    3. like distinct configure and build stages.
  3. We haven't covered the specifics of how autobuild integrates with the windows Visual Studio build system.
  4. We haven't covered specifically what the ABI requirements are for packages intended to be linked into the viewer.

Simple Example

It is also useful to look at the tiniest possible example of autobuild without the added complexity of building a large project like the Second Life Viewer, which has additional interacting build scripts, CMake and other build time elements that can obscure the picture of what autobuild is doing. To that end, there is a toy autobuild project here: Hello Autobuild which can be checked out and played with to get a feel for autobuild. Please do not check back changes to this repo so that everyone gets the same baseline to work from.