Modified DIY Linux
Reference Build

Note

This
is an unofficial and "unauthorized" version of the Reference Build
taken from here, as modified by myself, Jon Grosshart. I've
removed, added and adusted pieces of the document for an up-to-date
build. My ultimate goal is to document my slightly modified DIY build
using pkgtools in the same easy to read format that Greg uses, eg - this
web page and CSS. If this is a problem, please send me an e-mail and I
will remove it at once.

Permission is granted to copy, distribute, and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
copy of the license is available at
http://www.gnu.org/licenses/fdl.html.

This document may be copied and distributed in any medium, either
commercially or noncommercially, provided that the GNU Free
Documentation License (FDL), the copyright notices, and the license
notice saying the GNU FDL applies to the document are reproduced in all
copies, and that you add no other conditions whatsoever to those of the
GNU FDL.

1. Getting
Started

1.1. Environment

The very first task is to set up a sane environment in which to perform
the build. These environmental settings apply to the initial Temptools
phase, though most will also be carried through into the Chroot phase.

Decide where you want to build and install the system to. This doesn't
have to be a partition, it can be anywhere, under $HOME, /mnt, /tmp or
wherever you like, as long as there is plenty of space there (FIXME how
much?). The location you've chosen to build the system will be assigned
to the $SYSROOT variable below. Once the build has finished, the entire
root file system image may be transferred to its intended final
destination.

The following steps are taken to ensure a sane shell environment in
which to work in. Your build script should, in practice, inherit these
settings but it would be wise to at least sanity check the variables
yourself when writing your script.

Important

Be sure to substitute the location you've chosen for the build in the
$SYSROOT variable assignment below. You'll likely also want to amend
the value of $TZ to match your local time zone. $DIY_TARGET is used
to
select the target architecture you are building for. It is
vitally important to ensure that the
vendor field of the target triplet is something
other than "pc" or "unknown". This
is
because we want to force the pass 1 toolchain into cross compilation
mode. For example, if your real target is i686-pc-linux-gnu, set
$DIY_TARGET to i686-diy-linux-gnu or even i686-FAKE-linux-gnu. If
your
real target is x86_64-unknown-linux-gnu, set $DIY_TARGET to
x86_64-diy-linux-gnu, and so forth. Feel free to tweak $TT_PFX if you
don't care for the name "temptools". If you are building on SMP,
uncomment the MAKEFLAGS line to take advantage of make's parallel
execution feature. This is becoming more important as
multi-core
CPUs increase in popularity. Even on UP you can achieve quicker
builds
with export MAKEFLAGS="-j2" (at the
expense of
system responsiveness). If you are targeting x86_64 and also want the
ability to compile and run 32-bit code (recommended) set $BIARCH to
YES. This will provide a basic bi-arch (multilib) setup with 2
separate Glibc installations and the ability to compile the Grub
bootloader which depends on 32-bit compilation. From this basic
multilib setup, you then have the capability to build up a full-blown
multilib development environment (not covered here), but be aware
that
going down this path can be a lot of trouble for your average DIY'er.
Everything else should remain as is.

Note

I, Jon Grosshart,
build as the root user on the host the entire way through. For both
temptools and chroot. I've removed all refrences to the user 'build'. If
it wasn't clear from the very top of this document, I'm taking the
refrence build docs and modifying them the way I build. I've also
ditched pacman, fakeroot and everything else towards the end along with
the optional PPC packages.

I only build for x86 and x86_64 so that's all I'm leaving in.
Make sure the below environment commands are correct for your setup.

2. Temptools Phase

2.1. Bash-4.1
Pass 1

Important

This first pass of Bash is the only package we install "by hand".
Everything else after this point is expected to be automated via a
build script. However, you can still drive the build manually by
typing in all the commands on the interactive shell command line if
you prefer. Note: Do not skip Bash
Pass 1. It used to be optional but is now a requirement of the build
method due to our use of CONFIG_SHELL (see below).

Installing Bash as the very first package allows us to specify the
"configure shell" to be used by all Autoconf-produced configure
scripts during the Temptools phase. This is achieved via the
CONFIG_SHELL environment variable mentioned earlier. It serves to
enhance the robustness of the build method by reducing reliance on
the host's /bin/sh thus avoiding potential
breakage
when bootstrapping from older distros.

Another reason for installing Bash first up is that we can write
our
build script using all the features of modern Bash without having
to
worry about which shell is installed as /bin/sh on the host. Simply
launch your build script like this: `bash-pass1 myscript.sh'.

Tip

There is additional benefit to using this technique in that it
allows greater flexibility for handling the transition between
the
Temptools and Chroot phases. For example, you can set the
interpreter path in your build script to `#!/temptools/bin/bash'
which allows easy re-execution of the same script upon entering
the chroot environment without having to fuss over having a
`$SYSROOT/bin/sh' symlink already set up. Please refer to the
author's own
gsbuild
scripts for a sample implementation. Naturally, this point
is moot if you script in pure Bourne or handle the
Temptools/Chroot phases transition using a different method.

For our purposes of bootstrapping a new Linux system within the
Reference Build context, Bash should build fine on any sane distro
made within the last 5 years.

The "unset CONFIG_SHELL" in the configure invocation is needed for
configure to succeed because the shell pointed to by CONFIG_SHELL
doesn't exist yet i.e. we are installing it here.

Note: some older hosts have `libtermcap' installed and Bash will
prefer it over Ncurses if found by configure unless `--with-curses'
is given. Bash also comes with its own Termcap. Rather than tinker
with configure switches, it's actually more robust to just let
configure work it out for itself. This Pass 1 of Bash is simply for
running scripts during the Temptools phase so it shouldn't really
matter which Termcap is used.

Double ampersands are used here because we're still operating in an
interactive shell session at this point. We recommend that you
employ a proper error trapping strategy in your build script (eg:
set -e). Therefore, double ampersands are not provided in the build
commands from this point onwards.

2.2. Make-3.81 Pass 1

./configure
make
cp -v make ${TT_PFX}/bin
ln -sv make ${TT_PFX}/bin/gmake

Rationale Notes

for bootstrapping purposes, Make should build fine on any sane
distro made from 1999 onwards.

the Glibc build needs a version of `make' 3.79 or newer which older
distros don't have. We remove all doubts by installing the current
version. Why install it now rather than just prior to the Glibc
build? Well, it makes sense to integrate it from the outset so that
we can start using it immediately.

the `gmake' symlink is needed for robustness. `gmake' already
exists
on some hosts which will prevent Glibc's configure script from
finding the `make' we install here.

2.3. Sed-4.2.1 Pass 1

./configure
make
cp -v sed/sed ${TT_PFX}/bin

Rationale Notes

for bootstrapping purposes, Sed should build fine on any sane
distro
made from 1999 onwards.

Why do we install a Pass 1 of Sed here? There have been reported
problems where the Glibc build chokes with older Sed versions. The
author hasn't personally seen any such problem but feels it's safer
to just install a current Sed from the outset. An added bonus is
that we can rely on using `sed -i'
during the
entire Temptools phase thus simplifying scripting.

The `echo "MAKEINFO = :" >> Makefile'
tweak is used to remove dependence on `makeinfo' and therefore
eliminate all associated problems. Please read the mailing list
thread starting with
this
post for details.

We create a lib64 -> lib symlink in the x86_64 case because we
are not building a multi-arch toolchain. Even in single-arch mode
64-bit architectures tend to hardwire references to "lib64" in many
places throughout the toolchain. It's possible to force everything
to "lib" but we'd rather keep all the build commands compatible
with
other architectures. The symlink keeps things sane and everything
Just Works™.

After the first Binutils are installed, -B/usr/bin/ is temporarily
added to the CC definition in order to force the host GCC to use
the
host Binutils. Without this override, the host GCC will find the
newly installed Binutils in the PATH thus causing a toolchain
mismatch and potential breakage. This breakage is likely to strike
when using a bleeding edge distro as a host (eg: Fedora) and you're
building a toolchain comprised of older versions of toolchain
components than those on the host. The "if..echo..grep" statement
is
required on those hosts with GCC-2.95.x or earlier installed. More
background on all of this can be found in the mailing list thread
starting with
this
post (continued
here).

Note: Unlike LFS, we don't statically link the Pass 1 of Binutils
because it's simply unnecessary, and in actual fact is a potential
source of failure. We therefore link dynamically for robustness
reasons. If you still think static linking is a good idea then at
least take note of the Glibc maintainer's
views
on the subject.

Because we're linking dynamically, there's no need to specify `make
configure-host' after the initial configure. However, when building
on SMP the top level Makefile will run the sub-configure scripts
for
Libiberty and Binutils simultaneously. This is great for speed and
efficiency, but unfortunate for the build log which ends up all
jumbled. If you'd like the build log to remain neat and tidy (eg:
for diffing purposes), run the sub-configure scripts in serial by
executing make configure-host -j1
after the
initial configure.

GCC-4.3 and above requires the MPFR and GMP libraries. Here we are
taking advantage of the largely undocumented "combined tree"
feature
of the GNU toolchain. Integrating MPFR and GMP into the GCC build
itself greatly simplifies matters and mostly removes all host
issues
and other complications of building these libs separately. More
information in
this
mailing list post.

The switches `--disable-libmudflap', `--disable-libssp',
`--disable-libgomp' and `--disable-decimal-float' first appeared in
GCC-4.0, GCC-4.1, GCC-4.2 and GCC-4.3 respectively. They are used
here to streamline our initial "bootstrap" GCC. The switches are
all
compatible within supported GCC versions ie: earlier versions
simply
ignore the later switches.

Note: Starting with GCC-4.2, the top-level bootstrap mechanism
became the default. This means a plain "make" will now default to
performing a 3 stage bootstrap unless `--disable-bootstrap' is
given. The pre-GCC-4.2 method of "make bootstrap" behaves
identically. Therefore, we can maintain compatibility with earlier
versions by continuing to use "make bootstrap".

The libgcc_eh.a symlink is needed to satisfy the upcoming Glibc
build. Please read
this
mailing list post for some rationale discussion.

2.8. Glibc-2.11.1

Proceed
with Caution!

Glibc is possibly the most critical piece of software you will
compile
in a base Reference Build. There is a high probability of something
going wrong. Be sure to heed the advice contained in
Section B, “Glibc General Advice”
before taking on this monster.
Note: the Glibc compiled here in the Temptools phase is merely a
"throw away bootstrap" Glibc, but please do read the aforementioned
advice anyway.

We need to fiddle with `-march=' in CFLAGS because as of Glibc-2.6,
i486 is the minimum supported configuration on x86. More
information
here.

`AUTOCONF=no' is used here for robustness. It simply prevents
unnecessary Autoconf invocations when the Makefiles detect a
`configure' file older than its respective `configure.in'. This is
usually the case when working with CVS snapshots, but it also
applies to release tarballs prior to glibc-2.3.5 when the issue was
finally addressed. These unnecessary Autoconf invocations have been
known to cause build failures on Debian hosts. An alternative
approach is to run `find . -name configure | xargs touch' in the
src
dir before running `configure'. Note: `--without-cvs' does not
address the issue.

`MAKEINFO=:' is used to prevent installation failures when the host
has an older `makeinfo'. Problems were first noticed with
Glibc-2.7.

The tweak to STANDARD_STARTFILE_PREFIX_{1,2} ensures there is zero
chance of any libs and startfiles being found on the host. More
information in
this
mailing list post. Some older information related to the
same issue can be found
here.

As mentioned earlier, we need to pass `--disable-bootstrap' here to
prevent GCC-4.2 and above from performing a 3 stage bootstrap. The
switch is ignored by earlier versions and is therefore compatible.

The notes about `make configure-host' from Binutils Pass 1 and GCC
Pass 1 also apply here. However, it gets a little more complicated
because we're also building Libstdc++. To achieve the same outcome
(ie: run sub-configures in serial for neat build logs under SMP),
the sequence needs to be like this:

make configure-host -j1
make all-host
make configure-target -j1
make all-target

When building with GCC-4.x we use a sed to add
`-fomit-frame-pointer' to the GCC Makefile to address a correctness
issue. Please read
this
mailing list post for some rationale discussion.

2.25. Texinfo-4.13a

2.26. Bash-4.1 Pass 2

The configure switch `--without-bash-malloc'
causes Bash to use the
system supplied malloc from Glibc rather than Bash's own malloc.
Please read this
mailing list post for some rationale discussion.

2.27. M4-1.4.13

Tip

It's worth noting the next 3 packages (M4, Bison and Flex) are not
strictly necessary in the Temptools phase when using an official FSF
Binutils release. However, they are mandatory
if using a release from HJL. The reason is that FSF releases are made
with a "make dist" style procedure whereby the generated files (from
Bison and Flex) are already included in the tarball thus removing the
requirement for Bison and Flex at Binutils build time (in the Chroot
phase). Our recommendation is to always build M4, Bison and Flex here
to ensure the build recipe remains compatible with both FSF and HJL
Binutils releases. There is another issue: if you skip M4, Bison and
Flex here it's likely you'll run into
ICA
trouble which will require an additional
hack
when building Bison in the Chroot phase.

./configure
make V=1
cp -v src/m4 ${TT_PFX}/bin

2.28. Bison-2.4.1

./configure
make
make install

2.29. Flex-2.5.35

2.30. E2fsprogs-1.41.9
(SKIP)

mkdir build
cd build
../configure
make libs -j1
make install-libs

Rationale Notes

With versions of util-linux-ng ≤ 2.14.x, we would install the
E2fsprogs libraries here to satisfy the build of
`mount' in the next step. The trend has always been to install the
shlibs from e2fsprogs but now that util-linux-ng is shipping them, I'm
not entirely sure what to do (with regards to the chroot build). As far
as the temptools are concerned, I see no reason to install e2fsprogs
anymore.

The sed to `configure' forces Expect to call a plain `stty' instead
of `/bin/stty' or
even
worse, `/usr/local/bin/stty'. In this way the first `stty'
in the PATH will always be used. It also avoids the need to create a
`/bin/stty' symlink in Section 4.5, “Create Symlinks”.

We can remove the 'cross compile' toolchain
directories along with the unnecessary info, man and doc pages. It would
be wise at this point to tar up your temptools/ directory for safe
keeping.

3. Chroot Phase

3.1. Enter Chroot
Environment

Important

Before entering the chroot
environment, be sure to become root using
plain `/bin/su'. DO NOT use `/bin/su
-' (ie: with the hyphen) because you'll lose the
environment variables that are critical for the following chroot
command to work as intended. When including this chroot command in
your script you'll need to make 2 adjustments: (a) you should omit
the
$PS1 variable because it is normally unset in non-interactive shells
and (b) you'll want to execute a script instead of
${TT_PFX}/bin/bash --login +h (make
sure your
script does a set +h to switch off
hashing).

3.7. Pkgtools-13.0
Pass 1

Again, we have to manually create this package and
extract it to the root of our system before we can start making
packages with pkgtools. We won't bother installing our final pkgtools
until we build the ncurses package later on. Right now, this pkgtools
has linked against the ncurses in /temptools and is only temporary.

3.8. Makedev-1.7

cd /dev && ./MAKEDEV -v generic-nopty

Rationale Notes

Even tho we will be using udev to boot our system,
we'll need a compliment of device nodes during chroot. If not, test
suite failures will be common and things will just start to break in
general.

3.9. Base-8.0
Pass 2

sh base.build
installpkg /tmp/base-8.0-noarch-1.t*

Rationale Notes

Since we have extracted the etc-8.0 package which contains the file
/etc/group, the proper permissions will be set for /var/run/utmp and
/var/log/lastlog in the base-8.0 package... Also, now that we have a
preliminary pkgtools on the system, this will be our first 'official'
package made and installed with pkgtools. Go ahead... Type 'pkgtool' and
bask in the glory of your one whole package installed.