User:Henri Beauchamp/Automated Linux Build Script (1.20 and earlier)

From Second Life Wiki
Jump to navigation Jump to search

Return to Compiling the viewer (Linux)

Automated libraries and headers adjustments, compilation and packaging


Here is a bash scripts (for v1.20 and earlier viewers: for v1.21 and later, please see cmake-SL) that could save you a lot of time... It basically does all what is needed to automatically build and package a viewer in Linux.

IMPORTANT NOTE:

This script includes instructions which have the potential to remove MANY files on your system should you modify it without understanding exactly what it does (i.e. rm -rf $VARIABLE/ where entire portions of your filesystem *will* be deleted without warning).

Although some basic checks are done to avoid removing an entire home directory, or even the root tree, this script is not meant to be tweaked without a very good knowledge of bash scripting. It is therefore highly recommended that you use it without modifying the PATH_TO_SOURCES and TEMP_BUILD_DIR variables (modifying the USE_* and TUNE_FLAGS variables is safe).


#!/bin/bash

# make-SL v1.88 (c)2007-2009 Henri Beauchamp. Released under GPL license v2:
# http://www.gnu.org/licenses/gpl.txt

###############################################################################
######## THIS IS QUICK'N DIRTY ALPHA SOFTWARE. USE AT YOUR OWN RISKS ! ########
###############################################################################

###############################################################################
# BEWARE: this script is meant to compile a -personal- SL client. It is -NOT- #
#         suitable to build client versions meant for public release, because #
#         non-open source code is packaged by this script (e.g. libkdu).      #
###############################################################################

# This bash script is aimed at easying up the build process of a SL client.
# It does not cover building it with custom/system openjpeg, elfio or fmodapi
# (we use the provided libraries).
# You may enable or disable the use of your system's library by editing
# the USE_SYSTEM_* variable ("yes" --> use the system library, "no" --> use
# LL's provided ones).
# The script also takes care of updating properly the viewer_manifest.py script
# accordingly, so that you (should) end up with a properly packaged client.

# To use this script, simply make it executable (chmod +x make-SL) and
# put it into /usr/local/bin (or any other directory in your PATH).
# Then, download the slviewer-src-*.tar.gz, slviewer-linux-libs-*.tar.gz,
# slviewer-artwork-*.zip and fmodapi*.tar.gz archives, and finally, invoke
# make-SL as follow:
#   make-SL path_to_archives  (example: make-SL ~/downloads)
# or simply:
#   make-SL
# when invoking from the directory where the archives are.
# The sources will be installed into the PATH_TO_SOURCES directory,
# and the client will be built into the TEMP_BUILD_DIR directory.
# The packaged build will be moved to your home directory.
# If you want to retry a compilation after fixing something manually and
# don't want make-SL to start all over again, overwriting everything,
# you may invoke it with the --retry option, like this:
#   make-SL --retry
# This script has been tested by the author, on a (very customized)
# Mandrake 10.2 distro, and a (no less customized but yet more standard)
# Mandriva 2007.1 distro.
# Tested with SL v1.17, v1.18, v1.19 & v1.20 sources.

# Where the sources of the client will be held (defaults to "./linden"):
PATH_TO_SOURCES="/usr/src/SL"
# Where to build the client:
export TEMP_BUILD_DIR="$PATH_TO_SOURCES/build"

USE_SYSTEM_GTK="no"
USE_SYSTEM_SDL="no"
USE_SYSTEM_SSL="no"
# Beware: libdb4 makes use of libapr, and libapr makes use of libexpat... so
# you should keep USE_SYSTEM_APR, USE_SYSTEM_DB4 and USE_SYSTEM_EXPAT in sync.
USE_SYSTEM_APR="no"
USE_SYSTEM_DB4="no"
USE_SYSTEM_EXPAT="no"
USE_SYSTEM_OGG="no"
USE_SYSTEM_ZLIB="no"
USE_SYSTEM_UUID="no"
# If your system libcurl does not use c-ares (non-blocking DNS calls), better
# using LL's provided library...
USE_SYSTEM_CURL="no"
USE_SYSTEM_VORBIS="no"
USE_SYSTEM_XMLRPC="no"
USE_SYSTEM_JPEGLIB="no"
# Enabling the system freetype2 prevents the client from using the SL fonts,
# or at least, this is what happens on my system (Mandrake 10.2 and Mandriva
# 2007.1 with PLF's freetype2).
USE_SYSTEM_FREETYPE2="no"
USE_SYSTEM_PERFTOOLS="no"

# You may add tune flags here, to optimize the code for your processor.
# Example, for an Athlon XP:
# TUNE_FLAGS="-march=athlon-xp"
# BEWARE: because of a bug in the v1.17.2.0 SConstruct file, you MUST keep at
# least an -O2 flag in the TUNE_FLAGS, to avoid getting a very slooooow client...
# This problem was apparently fixed in v1.17.3.0.
TUNE_FLAGS="-fomit-frame-pointer -frename-registers -ftree-vectorize -fweb -fexpensive-optimizations -march=i686 -msse -mfpmath=sse"
# Set this to "yes" to force vectorization use in the viewer code (only for
# processors with SSE or Altivec units, and only if you did enable them via
# the TUNE_FLAGS.
FORCE_VECTORIZE="yes"

# Whether to build with the browser login screen or not:
WITH_MOZILLA="yes"

# When using gcc v4.2 or later, do set this to yes...
ALLOW_WARNINGS="no"

########### functions definitions ###########

function check() {
    if [ "$1" == "yes" ] ; then
        if [ -f $2 ] ; then
            echo "Using the system $3..."
            return 0
        else
            echo "WARNING: system $3 requested but not available..."
        fi
    fi
    return 1
}

function update_manifest() {
    grep -v $1 $PATH_TO_SOURCES/indra/newview/viewer_manifest.py >$TEMP_BUILD_DIR/viewer_manifest.py
    mv -f $TEMP_BUILD_DIR/viewer_manifest.py $PATH_TO_SOURCES/indra/newview/viewer_manifest.py
    chmod +x $PATH_TO_SOURCES/indra/newview/viewer_manifest.py
}

function compile() {
    cd $PATH_TO_SOURCES/indra
    echo "Compiling the client into $TEMP_BUILD_DIR..."
    scons -j6 DISTCC=no BTARGET=client OPENSOURCE=no BUILD=releasefordownload CHANNEL=Release MOZLIB=$WITH_MOZILLA MOZLIB2=$WITH_MOZILLA
    if (($? == 0)) ; then
        mv $PATH_TO_SOURCES/indra/newview/SecondLife*.tar.bz2 $HOME/
    fi
}

########### end of functions ###########

if [ "$TEMP_BUILD_DIR" == "" ] ; then
    export TEMP_BUILD_DIR=/tmp/$USER/SL
fi

# Check to see if we simply want to retry a compilation:
if [ "$1" == "--retry" ] ; then
    compile
    exit $?
fi

# Make sure we don't unpack over an old source tree:
if [ -d linden ] ; then
    rm -rf linden/
fi

if [ -d $PATH_TO_SOURCES ] && [ "$PATH_TO_SOURCES" != "" ] && [ "$PATH_TO_SOURCES" != "/" ] && [ "$PATH_TO_SOURCES" != "." ] && [ "$PATH_TO_SOURCES" != ".." ] && [ "$PATH_TO_SOURCES" != "$HOME" ] ; then
    rm -rf $PATH_TO_SOURCES/
fi

# Use the parameter (if any) as the path to the archives:

PATH_TO_ARCHIVES=`pwd`
if [ "$1" != "" ] && [ "$1" != "--prep" ]; then
    if [ -d $1 ] ; then
        PATH_TO_ARCHIVES=$1
	shift
    fi
fi

# Let's first unpack everything:
if ! [ -f $PATH_TO_ARCHIVES/slviewer-src-*.tar.gz ] ; then
    echo "You need slviewer-src !"
    exit 1
fi
if ! [ -f $PATH_TO_ARCHIVES/slviewer-linux-libs-*.tar.gz ] ; then
    echo "You need slviewer-linux-libs !"
    exit 1
fi
if ! [ -f $PATH_TO_ARCHIVES/slviewer-artwork-*.zip ] ; then
    echo "You need slviewer-artwork !"
    exit 1
fi
if ! [ -f $PATH_TO_ARCHIVES/fmodapi*.tar.gz ] ; then
    echo "You need fmodapi !"
    exit 1
fi
echo "Extracting the files from the archives..."
tar xzf $PATH_TO_ARCHIVES/slviewer-src-*.tar.gz
tar xzf $PATH_TO_ARCHIVES/slviewer-linux-libs-*.tar.gz
unzip -o $PATH_TO_ARCHIVES/slviewer-artwork-*.zip >/dev/null
# You may add artwork patches named in the form:
# slviewer-whatever-artwork-patch.zip
# Example: slviewer-v11900-statusbar-artwork-patch.zip
ARTWORK_PATCHES=`/bin/ls $PATH_TO_ARCHIVES/slviewer-*-artwork-patch.zip`
if [ "$ARTWORK_PATCHES" != "" ] ; then
    for f in $ARTWORK_PATCHES; do
        echo "Extracting artwork patch $f..."
        unzip -o $f >/dev/null
    done
fi

# fmodapi:
echo "Copying fmodapi files..."
tar xzf $PATH_TO_ARCHIVES/fmodapi*.tar.gz
cp -a fmodapi*/api/inc/* linden/libraries/i686-linux/include/
cp fmodapi*/api/libfmod-*.so linden/libraries/i686-linux/lib_release_client/
rm -rf fmodapi*/

# Move the sources to where we want to hold them:
if [ "$PATH_TO_SOURCES" != "linden" ] && [ "$PATH_TO_SOURCES" != "" ] ; then
    echo "Moving the sources to $PATH_TO_SOURCES..."
    mv -f linden $PATH_TO_SOURCES
fi

# Let's own the files:
chown -R $USER: $PATH_TO_SOURCES/

# Check for patches to apply. The names of the patches must be in the form
# slviewer-*.patch* (Example: slviewer-v117-EmbbededNotecard.patch.bz2).
# They must be applicable from inside the source directory with the -p1
# option, i.e. they have been built from outside the source directory
# with a diff command such as:
# diff -urN linden/ linden-patched/ >slviewer-whatever.patch
# And they may be gzipped or bzipped.
PATCHES=`/bin/ls $PATH_TO_ARCHIVES/slviewer-*.patch* 2>/dev/null`
if [ "$PATCHES" != "" ] ; then
    echo "Applying patches..."
    cd $PATH_TO_SOURCES
    for i in $PATCHES; do
        echo "Patch: $i"
        if echo $i | grep ".gz" &>/dev/null ; then
            gunzip -c $i | patch -p1 -s
        elif echo $i | grep ".bz2" &>/dev/null ; then
            bzcat $i | patch -p1 -s
        else
            patch -p1 -s <$i
        fi
    done
fi

# Make a clean build:
if [ -d $TEMP_BUILD_DIR ] && [ "$TEMP_BUILD_DIR" != "" ] && [ "$TEMP_BUILD_DIR" != "/" ] && [ "$TEMP_BUILD_DIR" != "." ] && [ "$TEMP_BUILD_DIR" != ".." ] && [ "$TEMP_BUILD_DIR" != "$HOME" ] ; then
    rm -rf $TEMP_BUILD_DIR/
fi
mkdir -p $TEMP_BUILD_DIR

# In v1.18.4.0, they use c-ares but fail to provide it in the library package...
if [ -f $PATH_TO_SOURCES/indra/llcommon/llares.cpp ] ; then
    if ! [ -f $PATH_TO_SOURCES/libraries/i686-linux/include/ares/ares_dns.h ] && ! [ -f $PATH_TO_SOURCES/libraries/include/ares/ares_dns.h ] ; then
        if [ -f /usr/include/ares_dns.h ] && [ -f /usr/lib/libcares.a ] ; then
            echo "Copying missing c-ares files from the system..."
            cd $PATH_TO_SOURCES/libraries/i686-linux
            mkdir include/ares
            cp -a /usr/include/ares* include/ares/
            cp -a /usr/lib/libcares* lib_release_client/
        else
            echo "Sorry... You need c-ares installed in your system to compile this version of the client."
            echo "Try: http://daniel.haxx.se/projects/c-ares/"
            exit 1
        fi
    fi
fi

# Let's use the system GTK+ if available:
if check $USE_SYSTEM_GTK /usr/include/atk-1.0/atk/atk.h "GTK+" ; then
    cd $PATH_TO_SOURCES/libraries/i686-linux/lib_release_client
    rm -f libgtk* libgdk* libglib* libgmodule* libgobject* libgthread* libpango* libatk*
    cd ../include
    rm -rf atk-1.0/ gtk-2.0/ glib-2.0/ pango-1.0/
    cp -a /usr/include/atk-1.0 .
    cp -a /usr/include/gtk-2.0 .
    cp -a /usr/lib/gtk-2.0/include/* gtk-2.0/
    cp -a /usr/include/glib-2.0 .
    cp -a /usr/lib/glib-2.0/include/* glib-2.0/
    cp -a /usr/include/pango-1.0 .
    if [ -d /usr/include/cairo ] ; then
        cp -a /usr/include/cairo/* .
    fi
fi

# Let's use the system freetype2 if available:
if check $USE_SYSTEM_FREETYPE2 /usr/include/ft2build.h "freetype2" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/freetype/ i686-linux/include/llfreetype2/*
    rm -f include/ft2build.h i686-linux/lib_release_client/libfreetype.a
    cp -a /usr/include/freetype2/freetype/ i686-linux/include/llfreetype2/
    cp -a /usr/include/ft2build.h i686-linux/include/llfreetype2/freetype/
fi

# Let's use the system zlib if available:
if check $USE_SYSTEM_ZLIB /usr/include/zlib.h "zlib" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/zlib/
    mkdir -p i686-linux/include/zlib
    cp -a /usr/include/zlib*.h i686-linux/include/zlib/
fi

# Let's use the system jpeglib if available:
if check $USE_SYSTEM_JPEGLIB /usr/include/jpeglib.h "jpeglib" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/jpeglib/ i686-linux/lib_release_client/libjpeg.a
    mkdir -p i686-linux/include/jpeglib
    cp -a /usr/include/j*.h i686-linux/include/jpeglib/
    touch i686-linux/include/jpeglib/jinclude.h
fi

# Let's use the system xmlrpc-epi if available:
if check $USE_SYSTEM_XMLRPC /usr/include/xmlrpc.h "xmlrpc-epi" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/xmlrpc-epi/ i686-linux/lib_release_client/libxmlrpc.a
    mkdir -p i686-linux/include/xmlrpc-epi
    cp -a /usr/include/xmlrpc*.h i686-linux/include/xmlrpc-epi/
fi

# Let's use the system ogg if available:
if check $USE_SYSTEM_OGG /usr/include/ogg/ogg.h "ogg" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/ogg/ i686-linux/lib_release_client/libogg*
    update_manifest libogg
fi

# Let's use the system vorbis if available:
if check $USE_SYSTEM_VORBIS /usr/include/vorbis/vorbisenc.h "vorbis" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/vorbis/ i686-linux/lib_release_client/libvorbis*
    update_manifest libvorbis
fi

# Let's use the system SDL if available:
if check $USE_SYSTEM_SDL /usr/include/SDL/SDL.h "SDL" ; then
    cd $PATH_TO_SOURCES/libraries/i686-linux
    rm -rf include/SDL/ lib_release_client/libSDL*
    update_manifest libSDL
fi

# Let's use the system openssl if available:
if check $USE_SYSTEM_SSL /usr/include/openssl/opensslconf.h "openssl" ; then
    cd $PATH_TO_SOURCES/libraries/i686-linux/lib_release_client
    rm -f libssl.* libcrypto.*
    update_manifest libssl
    update_manifest libcrypto
fi

# Let's use the system apr if available:
if check $USE_SYSTEM_APR /usr/include/apr*/apr_base64.h "apr" ; then
    cd $PATH_TO_SOURCES/libraries/i686-linux
    rm -rf include/apr-1/*
    rm -f lib_release_client/libapr*
    cp -a /usr/include/apr*/* include/apr-1/
    update_manifest libapr
fi

# Let's use the system db4 if available:
if check $USE_SYSTEM_DB4 /usr/include/db4/db.h "db4" ; then
    rm -f $PATH_TO_SOURCES/libraries/i686-linux/lib_release_client/libdb*.so
    update_manifest libdb
fi

# Let's use the system expat if available:
if check $USE_SYSTEM_EXPAT /usr/include/expat.h "expat" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/expat/
    rm -f i686-linux/lib_release_client/libexpat*
    mkdir -p i686-linux/include/expat
    cp -a /usr/include/expat*.h i686-linux/include/expat/
    update_manifest libexpat
fi

# Let's use the system curl if available:
if check $USE_SYSTEM_CURL /usr/include/curl/curl.h "curl" ; then
    cd $PATH_TO_SOURCES/libraries
    rm -rf include/curl/
    rm -f i686-linux/lib_release_client/libcurl.*
    update_manifest libcurl
fi

# Let's use the system uuid if available:
if check $USE_SYSTEM_UUID /lib/libuuid.so.1 "libuuid" ; then
    rm -f $PATH_TO_SOURCES/libraries/i686-linux/lib_release_client/libuuid.*
    update_manifest libuuid
fi

if grep tcmalloc $PATH_TO_SOURCES/indra/SConstruct &>/dev/null ; then
    if check $USE_SYSTEM_PERFTOOLS /usr/include/google/malloc_hook.h "google-perftools" ; then
        cd $PATH_TO_SOURCES/libraries/i686-linux
        rm -f lib_release_client/libtcmalloc.* lib_release_client/libstacktrace.*
        rm -rf include/google/
        cp -a /usr/lib/libstacktrace.* /usr/lib/libtcmalloc.so* lib_release_client/
        cp -a /usr/include/google include/
        update_manifest tcmalloc
        update_manifest stacktrace
    else
        # Remove the google-perftools stuff if not present in the libraries package
        # (problem seen in SL v1.17):
        if ! [ -f $PATH_TO_SOURCES/libraries/i686-linux/lib_release_client/libtcmalloc.so ] ; then
            update_manifest tcmalloc
            update_manifest stacktrace
            # The following lines remove the
            #   "external_libs += ['tcmalloc', 'stacktrace']"
            # line from SConstruct... and many other lines which renders SConstruct
            # unusable to build anything else than the client... this is a quick'n
            # dirty patch. :-P
            grep -v tcmalloc $PATH_TO_SOURCES/indra/SConstruct >$TEMP_BUILD_DIR/SConstruct
            mv -f $TEMP_BUILD_DIR/SConstruct $PATH_TO_SOURCES/indra/SConstruct
        fi
    fi
fi

# Fix a bug in v1.17.1 and later SConstruct file, where some libraries are
# compiled as shared instead of static, and are yet not packaged, and where
# fmod can't be enabled for OPENSOURCE:
sed -e "s/create_dynamic_module(module=module, module_libs=module_libs)/create_static_module(module)/" $PATH_TO_SOURCES/indra/SConstruct >$TEMP_BUILD_DIR/SConstruct
sed -e "s/enable_fmod = not opensource and/enable_fmod =/" $TEMP_BUILD_DIR/SConstruct >$PATH_TO_SOURCES/indra/SConstruct
rm -f $TEMP_BUILD_DIR/SConstruct

# Force the vectorization use if we chose so.
if [ "$FORCE_VECTORIZE" == "yes" ] ; then
    TUNE_FLAGS="$TUNE_FLAGS -DLL_VECTORIZE=1"
fi

# Add tune flags, if any:
if [ "$TUNE_FLAGS" != "" ] ; then
    sed -e "s/-pipe/$TUNE_FLAGS -pipe/" $PATH_TO_SOURCES/indra/SConstruct >$TEMP_BUILD_DIR/SConstruct
    mv -f $TEMP_BUILD_DIR/SConstruct $PATH_TO_SOURCES/indra/SConstruct
fi

# Use whatever gcc version we got.
sed -e "s/g++-3.4/g++/" $PATH_TO_SOURCES/indra/SConstruct >$TEMP_BUILD_DIR/SConstruct
sed -e "s/g++-4.1/g++/" $TEMP_BUILD_DIR/SConstruct >$PATH_TO_SOURCES/indra/SConstruct
rm -f $TEMP_BUILD_DIR/SConstruct

# Fix boost library linking problems in v1.20
sed -e "s/-gcc34-mt//g" $PATH_TO_SOURCES/indra/SConstruct >$TEMP_BUILD_DIR/SConstruct
mv -f $TEMP_BUILD_DIR/SConstruct $PATH_TO_SOURCES/indra/SConstruct

# Newer GCCs (4.2 and later) will issue many warnings: let's disable -Werror
if [ "$ALLOW_WARNINGS" == "yes" ] ; then
    sed -e "s/-Werror //" $PATH_TO_SOURCES/indra/SConstruct >$TEMP_BUILD_DIR/SConstruct
    mv -f $TEMP_BUILD_DIR/SConstruct $PATH_TO_SOURCES/indra/SConstruct
fi

# Update the manifest file and enable libkdu packaging. I'm too lazy to use
# awk or perl, here, but that's indeed what should be done...
# The following (very dirty) code may break at some point, depending on what
# LL will put in comments in the manifest file...
# First, remove the lines with libstdc++.so.6 and the crash logger:
update_manifest libstdc
update_manifest crash_logger
# Now, any line with a '#' followed with several spaces _should_ be dealing
# with the libkdu stuff... So, we simply remove the '#"...
sed -e "s/#        them/# them/" $PATH_TO_SOURCES/indra/newview/viewer_manifest.py >$TEMP_BUILD_DIR/viewer_manifest.py
sed -e "s/#   /   /" $TEMP_BUILD_DIR/viewer_manifest.py >$PATH_TO_SOURCES/indra/newview/viewer_manifest.py
rm -f $TEMP_BUILD_DIR/viewer_manifest.py
chmod +x $PATH_TO_SOURCES/indra/newview/viewer_manifest.py

# Comment out "export LL_GL_BASICEXT=x" in the wrapper script:
sed -e "s/export LL_GL_BASICEXT=x/#export LL_GL_BASICEXT=x/" $PATH_TO_SOURCES/indra/newview/linux_tools/wrapper.sh >$TEMP_BUILD_DIR/wrapper.sh
mv -f $TEMP_BUILD_DIR/wrapper.sh $PATH_TO_SOURCES/indra/newview/linux_tools/wrapper.sh
chmod +x $PATH_TO_SOURCES/indra/newview/linux_tools/wrapper.sh

# Missing file... at least in v1.17.0.0
if grep "secondlife-i686.supp" $PATH_TO_SOURCES/indra/newview/viewer_manifest.py &>/dev/null ; then
    if ! [ -f $PATH_TO_SOURCES/indra/newview/secondlife-i686.supp ] ; then
        touch $PATH_TO_SOURCES/indra/newview/secondlife-i686.supp
    fi
fi

# LL makes the assumption that a certain unicode true type font is installed
# on all Linux systems... which is of course wrong !  Let's check, and when
# this font is absent, replace the broken soft link with an empty file: this
# at least will prevent the viewer to issue many warnings about not being able
# to load the unicode.ttf font.
if ! [ -r $PATH_TO_SOURCES/indra/newview/linux_tools/unicode.ttf ] ; then
    rm -f $PATH_TO_SOURCES/indra/newview/linux_tools/unicode.ttf
    touch $PATH_TO_SOURCES/indra/newview/linux_tools/unicode.ttf
fi

# Check to see if we simply want to unpack the sources:
if [ "$1" == "--prep" ] ; then
    echo "Prepared sources available in: $PATH_TO_SOURCES"
    exit 0
fi

compile