Advanced Build Environment (ABE)

ABE is a tool for building the GNU toolchain for native or cross compilers. Building a functional toolchain involves many steps that have to be executed in the proper order with all the right command line arguments when configuring and building. Decades ago we used the "Cygnus Makefile", which was a top level Makefile used at Cygnus Support. After Cygnus was sold to RedHat a new utility popped up, Crosstool-ng, which is still actively maintained and used by many.

ABE has a different focus since it was designed to build official toolchain releases, including Windows hosted cross-compilers. ABE was also desighned to run within a Gerrit and Jenkins continuous build and test system to do CI on every submitted patch. Each release has a generated manifest file that lists all the verions of all the packages which can be used to later rebuild the exact configuration. Very useful when tracking down obscure bugs many months later.

People ask why use Bourne Shell ? With bourne shell there are no dependencies other than the standard Unix command line utilities, so it's very portable and stable for a long time. Also building the toolchain is a command line driven process, so Bourne shell works well with the GNU Autotools used by the toolchain.

Configuring ABE

Although it's a shell script, ABE does need to be configured with some build machine specific parameters. The configuration process checks for all the dependencies required to build a toolchain release, and creates the top level directories need to build the toolchaihn.

There are several directories that get created by configure.

* snapshots - This is where all the downloaded sources get
stored.
* builds - This is where all the executables get built and
installed.

The configure process produces a host.conf file with the default settings. This file is read by ABE at runtime, so it's possible to change the values and rerun abe.sh to use the new values.

Source Locations

The default URL to download source code is in the config/source.list file. For each component, the name is used as an index into the file. A component name with .git in it is used for a remote git repository. A component name with a .git in it is a tarball.

For example:

* gmp https://gcc.gnu.org/pub/gcc/infrastructure
* gcc.git git://gcc.gnu.org/git/gcc.git

Component Configuration

Each component in the toolchain has it's own configuration file with the default values which are contained in the config/ directory. These are also in Bourne Shell format. In these files host or target architecture specific configuration and build options can be set. There are a few default variables that can be set in each file.

latest

This is set to the default version of this component to build. It can refer to a tarball or a git repository. There is a convoluted syntax that can be used to specify tags, release versions, or git commit IDs.

For example:

* latest="gmp-6.3.0.tar.bz2" to use a tarball
* latest="gmp.git~master" to use a git branch
* latest="gmp.git@12345" to use a git commit ID

default_configure_flags

This variable is key to building a toolchain. In it any host and target specific values can be set. Other options from the build configuration are also available, for example which C ligbrary to use (glibc, newlib, bionic). Setting this variable can get quite complicated for GCC, most other componets are simpler.

default_makeflags

This optional variable sets build time options to make. By default make is invoked with no options. The number of parallel builds is based on the number of CPU cores in the build machine.

runtest_flags

This optional variable sets the options for running the tests, ala make check. By default each component has the defaults for running texts, but this can be used to add debugging flags when tracking down test failures.

GCC Stages

When built natively, GCC only builds itself once, and is fully functional as the sysroot is part of the OS. When cross compiling, this works differently. A fully functional GCC can't be built without a sysroot, so a minimum compiler is built to build the C library. This is called stage 1. This is then used to compiler the C library. Once this library is installed, then stage 2 of GCC can be built, which includes all the other front ends like g++, or go. By default only the C, C++ and Fortran front ends are built. This can be changed at build time by specifying --set languages=c,c++,go,rust for exaple.

Building a Toolchain

Building a cross toolchain is copmplicated, and prone to breakage even with minor changes to a file. ABE was designed to support developers, not just building releases so there is fine grained control of eacch component. By default ABE tries to build everything, and if started in a freshly git cloned version of ABE, will download and build everything with no intervention. The current versions of all the software is used, although this can be changed. Come back a few hours later...

To build a cross compiler, all that needs to be specified is simple:

./abe.sh --target aarch64-none-elf --build all

Note that the none-elf target is for bare metal, so will default to newlib for the C libary. For Unix based platforms Glibc is used instead. There is experimental support for building the Android bionic C library.

Components

ABE calls each toolchain a component. This is used to refer to a specific program. The current list of components is: gmp, mpc, mpfr, binutils, gcc, gdb, glibc, newlib, and qemu.

Changing Component Versions

It is possible at build time to overrule the default version of a component. Any of the components can be redefined via the command line using the same syntax as used in the config files. For example here's how to build a specific revision of GDB, and only rebuild GDB.

./abe.sh --disable update --build gdb GDB="gdb.git@12345"

Other Options

There are many other options

* --release - specify the release version for the created tarball
* --target - the target architecture
* --host - the host architecture if hot the build machine (mingw64)
* --dryrun - Don't actually do anything
* --set - Many options for fine grained control
* --check - Run the testsuite

Downloading Source

ABE know how to download tarballs or clone git repositories. This is defined in the config/sources.conf file, and the component specific files in the config directory. While downloading all the software is the default, you can also download everything and not build it. By default everything is downloaded to the snapshots directory. For git repositories in the snapshots directory, you'll notice the sources are ohnly fetched, not checked out yet. That happens at build time.

./abe.sh --checkout all
./abe.sh --checkout gmp

Then later you can build a toolchain and disable downloading. Often during development you don't want to change any of the dependencies. To disable downloading add this option:

./abe.sh --disable update --target aarch64-linux-android --build all

At build time the version specified in the latest variable from the config file is used to checkout the sources from the downloaded repository. For example:

* mpfr-4.2.2.tar.bz2 becomes a mpfr-4.2.2/ directory
* gcc.git~master becomes a gcc.git~master/ directory.
* gdb.git@12345 becomes a gdb.git@12345/ directory.

Building a Release

ABE was designed to build officially supported releases of the GNU toolchain. It can produce a release tarball with everything statically linked so runs on most Unix platforms. For Windows hosts, a zip file is created. By default for development builds, the name is really long and contains the the commit ID or version and the host and target. An example devlopment release package would look like this:

gcc-unofficial-16.0.0-16.0.0-x86_64_x86_64-pc-linux-gnu.tar.xz

The release process generates both a binary tarball, and a source tarball.

This can be changed at runtime

* --tarbin - Build a binary tarball of executables
* --tarsrc - Build a source tarball
* --release - Specify the release version string

This will also generate a manifest file of eacch component in the release. This can also be read in by ABE to produce an exact clone of the toolchains.

Windows Host

Using Mingw64 it's possible to build exectables on a Unix machine that will run on MS Windows. This is called a Canadian Cross, named after the 3 party system in Canada by a Canadian developer. The GNU Autools has full support for this configuration.

./abe.sh --build all --target arm-none-elf --host x86_64-mingw64

Notes on Bourne Shell Scripting

These scripts use a few advance features that are Bash specific. One is heavy use of bourne shell functions to reduce duplication, and make the code better organized. Any string echo'd by a function becomes it's return value. Bourne shell supports 2 types of return values though. One is the string returned by the function. This is used whenever the called function returns data. This is captured by the normal shell commands like this: values=$(call_function 1ドル). The other type of return value is a single integer. Much like system calls, these scripts all return 0 for success, and 1 for errors. This enables the calling function to trap errors, and handle them in a clean fashion. Most functions return both, a sucess/failure code, and the string output. Often the string output is an associative array.

The other is heavy use of associative arrays, which is the only data structure Bourne shell supports. This enables a shell function to return complex data structures all grouped together.

A few good habits to mention, always enclose a sub shell execution in double quotes. If the returned string contains spaces, this preserves the data, otherwise it'll get truncated.

Another good habit is to always prepend a character when doing string comparisons. If one of the two strings is undefined, the script will abort. So always using "test x${foo} = xbar" prevents that.