crosstool-NG HOWTO & gotchas

cross-toolchains are typically used to build code that is supposed to run on
a machine with a different architecture, e.g. when developing embedded code
for an ARM-based box on a x86 dev machine. however, they’re also very useful
when you want to make sure that all the devs in a group use exactly the same
set of toolchain binaries and libraries regardless of which versions of gcc,
headers and libraries those devs have installed locally on their dev machines.

a cross-toolchain consists of a bunch of tools and libraries, which must play
well together if it’s ever to be useful. coming up with a specific set of
specific versions of tools/libs that will peacefully co-exist is not for
the faint at heart, especially if you have wide-ranging requirements and
want extra goodies (which you will want). it is even more of a PITA
if you want to build a static (and relocatable) cross-toolchain.

overview:

crosstool-NG (from here on “CT”) comes with a largish set of “Preconfigured
toolchains”, or “samples”. those, allegedly, were proven to work together to
one degree or another, and can provide inspiration (read: copy-paste) for
creating your own. to see the list of such samples:

ct-ng list-samples

to see the components (tools/libs) that make up such a sample and their
versions:

to see similar output for a toolchain you config yourself using menuconfig:

ct-ng show-config

after a number of false-starts, i arrived at the following mix of tools/libs
versions that are close to my requirements and can be built into a static
cross-toolchain on fedora-15, and seem to pass initial tests:

building the cross-toolchain:

$CT_INST - this is the --prefix you specified during ./configure,
where your installed CT will land, assuming non-root install.

$CT_ROOT - this is wherever you run ct-ng binary from. CT will do
everything under that dir, so make sure to run ct-ng inside a dedicated dir.

$CT_XTOOLS - root of the resultant cross-toolchains’ prefix dirs,
this is where the final toolchains land.

flip through the dialogs of:

ct-ng menuconfig

choosing the packages you want. for best results you’ll prolly want to enable
the EXPERIMENTAL stuff early on, otherwise most of the recent package
versions will not be available and a lot of useful cross-toolchain
functionality will be missing.

first things first:

“Paths and misc options” ->

“Prefix directory” - this is where the final cross-toolchain will
land, called $CT_XTOOLS in this document. it defaults under your
$HOME/x-tools, change as necessary.

“Number of parallel jobs” - set according to your number of cores/CPUs,
though help recommends to set it to TWICE the number of CPUs in my
experience that’s way to high.

“Maximum allowed load” - very handy, again, set according to the HW
you build on, leaving yourself some breathing space.

“Tuple’s vendor string” - the default is unknown. “target tuple” is
the combo: arch-VENDOR-kernel-system, e.g. powerpc-e300c3-linux-gnu.
this is the VENDOR bit of the tuple.

“Tuple’s alias” - this one’s handy, it generates short-named symlinks
to the same tools right alongside the tools themselves. e.g. if you
set it to acme and build powerpc-e300c3-linux-gnu tuple you’ll
end up with powerpc-e300c3-linux-gnu-gcc AND acme-gcc binaries under
bin.

the rest - per your target and requirements.

if you’re even slightly adventurous in your configuration, you’ll witness a
pile of failures before you make the darned thing build your cross-toolchain
to your liking, most of them trivial and obvious, some - well documented.
make sure to read:

first step in troubleshooting failed build is to open $CT_ROOT/build.log,
jump to the bottom and try to guess what went wrong. if it’s some failed
dependency that one of the packages’ ./configure script notices, you may have
to drill down to the specific config log to figure out what exactly went
wrong and made ./configure say that some thing or other is “missing”.
per-tool/lib logs land under:

$CT_ROOT/.build/<toolchain_name>/build/build-<package>/config.log

e.g.:

/tmp/x86_64-acme-linux-gnu/build/build-ppl/config.log

NOTE: CT sporadically fails to download packages from sourceforge, even
though they appear to be there. just peek at the log that gets created as:

$CT_ROOT/build.log

look up which package CT tried to download and failed, copy the address
and download it manually into:

$CT_ROOT/.build/tarballs

CT will notice the packages already there and use them when you re-run it.
strace & duma are especially prone to this at this time of the year…

if still in the experimentation stage, you’ll want to make sure you:

enable [EXTRA] traces to see at a glance where CT fails. you’ll want
“Paths and misc options” -> “Maximum log level to see” set to EXTRA
for that.

enable “Paths and misc options” -> “Debug crosstool-NG” and then under
it: “Save intermediate steps” (“gzip saved states” is up to you). rest
assured your build WILL fail due to dependencies for a while before you
nail it. unless you have this option on, when you’ll restart the build
again after fixing issues, CT will do everything FROM SCRATCH. with
this option on, after each tool/lib successfully built, CT will print (with
tool/lib name instead of <step-name>, of course):

Saving state to restart at step '<step-name>'...

and you’ll be able to restart from that specific LNG state by doing:

ct-ng <step-name>+

e.g.:

ct-ng libc+

YOU WANT THIS OPTION ON UNTIL YOU FIGURE OUT YOUR PERFECT SET OF TOOL/LIB
VERSIONS THAT WORKS. no, really, you do!

back up the tarballs that CT downloads in the process to the:

$CT_ROOT/.build/tarballs/

dir - they DO get wiped out in case of ct-ng distclean and CT WILL
take ages to re-download the entire >200MB again! stash them someplace read-
only, then specify this location under Paths and misc options” ->
“Local tarballs directory”.

read the help messages of various options.

you prolly will want to build a fully static cross-toolchain to make sure
it’ll run on any reasonably similar system (e.g. fedoras/RHELs/CentOSes) and
can be “deployed” by a simple untar without having to install any libs on
target or specify any paths to any .so libs (or just dumped on a central
server exporting NFS to be mounted from the dev machines - though you’ll pay
dearly in build times for that stunt!). to build even a moderately useful
static cross-toolchain you’ll probably need to install at least:

you can wait for build to fail case-by-case until it forces you to install
those, or just take my word for it and install them beforehand.

NOTE: recent versions of GDB require expat (WTF?), staticcross-gdb requires static libexpat.a, but for some reason, the expat
package on fedora does NOT provide a static lib, and unlike the above
*-static packages there’s NO expat-static package on fedora (but
there’s an open bug
in RH’s bugzilla, if that’s any consolation). although CT downloads expat
tarball, it ONLY builds a static version for the target, and even then,
during native gdb phase which comes AFTER the cross-gdb phase. Yann
refuses
to build the libexpat.a for HOST for ideological reasons so, just
./configure + make the same version of expat as CT grabs (just reuse
the same tar) and park the resultant libexpat.a under /usr/lib64.
GRRR!!!

sanity-checking your cross-toolchain:

once you’ve managed to build your cross-toolchain start-to-finish without
failures, it’s HIGHLY RECOMMENDED that you do some sanity checks on it. tick
off: “Companion libraries” -> “Check the companion libraries builds” and
“Test suite” -> “GCC test suite” in menuconfig, then re-start the build
from scratch. no need for distclean, but do a ct-ng build - restarting from
the middle using <stage>+ will NOT pick up the changes in .config!

the former option will run the tests during the build, so once the build is
successfully over - you’re good. it also takes oh-shit-long (on the order of
90 minutes on a 4 x 3.1GHz machine with gobs of RAM) which is why you DON’T
want to enable it by default…

the latter option just builds the GCC test suite, leaving it under:

$CT_XTOOLS/<toolchain_name>/test-suite/gcc/

this one is a bit of a beast: it’s really designed to TEST that a
cross-compiler works by running compiled test snippets on an actual target.
it’ll need dejagnu + expect on the host, and passwordless ssh connection to
a target. IF your host is able to run the target binaries, like a 64-bit ->
32-bit cross-compiler for the same arch, you can just specify localhost as
a target. luckily, i THINK CT pre-configures the test suite by putting the
right strings into its Makefile, so all you really need to do is:

(where luser is the username dejagnu will use to connect, q.v. passwordless
above). this will first compile a bunch of compile-only GCC regression tests on
your host, then enter a loop where on each iteration it’ll compile a single
compile+run regression test, scp it onto <tgt_addy> (possibly prompting you
with yes/no if it’s the 1st ssh connection to that host, so don’t leave
unattended if you want results!), then it’ll ssh to <tgt_addy> to run this
test, etc. finally, repeat for g++:

there’s a pretty good chance that in the tests summary you WILL see SOME
“unexpected failures” (likely the same several tests failing multiple times
with different compilation flags, inflating the reported failures numbers).
google up the circumstances on a per case basis: usually it boils down to bugs
already reported upstream in RH/GNU/sourceware bugzillas, some of them already
fixed upstream and possibly released in newer tools versions. MAKE SURE THAT
THIS IS THE CASE and that it’s NOT your particular cross-toolchain that is
broken, or you’ll find yourself debugging ghosts during actual development
using the resultant cross-toolchain!

for the sake of comparison, on my cross-toolchains i got results in the
vicinity of >20 “unexpected failures” for gcc, one or more “unexpected
failures” for g++, the latter often involving the linker.

known issues:

older glibc refuse to build with make v3.82: “mixed implicit and normal
rules”. just select “Companion tools” -> “Build some companion tools”
and select make which will build make v3.81. CT will use it happily from
there on.

ppl turned out to be a spiteful little thing, bearing a grudge against GMP
(and CT users, apparently):

PPL v0.11* configure script will ignore pretty much ANY version of GMP you
might select in CT config. just fall back to v0.10*: gmp-4.3.2 +
ppl-0.10.2 seem to work, but will probably necessitate downgrade of GCC
from v4.6.x to v4.5.x.