Emdebian distribution

Debian is a fantastic resource
with its 20,000 packages and 12 architectures, but it's too fat for a
lot of embedded and small-system use. Below we describe the emdebian
process for producing a version of Debian that is smaller, but retains
as much that is good about Debian as possible. The packaging system,
the licensing, the distributed development, the vendor neutrality,
the multi-architecture support, the guaranteed freedom, the
availability of source, the build system, the consistency
between packages.

Unfortunately we can't just change Debian policy to mandate smaller
packages, or allow the removal of important documentation, as that
wouldn't suit most Debian users. Yet simply taking a snapshot of Debian
and modifying the packages will mean our packages are different from
those in Debian and we will get left behind as Debian continues to
develop. One solution to this is support for "dpkg variants"
where changes needed to derive an Emdebian package can be fully
integrated into the Debian package.

We need a scheme that allows Emdebian to stay in sync with Debian
as much as possible, whilst having fine control over package generation.
To do this we need to keep Emdebian package modifications in each
package, maintained by package maintainers as much as possible. The
initial modifications will be done by the Emdebian team in the form
of new rules and control files in emdebian patch files. These
modifications are passed back to maintainers or hopefully can be
persuaded to keep them up to date for their packages.

The current composite method combines the best of previous methods
by using patches stored in subversion. For details of the latest version
of the composite method and details of previous methods, see the MetaData Wiki page.

This is a huge undertaking but fortunately we don't need to do this
for every package in Debian, only the relatively small subset that most
embedded/small systems need. Much of the process can be automated, at
least to a first approximation, and once the principle is proved we can
hopefully get the maintenance of the emdebian changes mandated as part
of Debian policy. However that's well in the future. Right now we need
to set up the system, make patches, persuade maintainers and persuade
ourselves that we have a practical and useful system.

The Composite Plan

This scheme has been worked out over a succession of meetings
during late 2006 by a number of project members, ably assisted by
interested Debian developers. It is loosely based around a combination
of the $DEBIAN_DIR and udebs methods. It has also been greatly
improved by discussion on the mailing list. Ultimately
Wookey and Neil Williams are to blame for any misunderstandings
in committing it to text. So if you think it's rubbish you know who to
blame. Criticism is welcome and changes may well need to be made as we
try things out.

Initially, we will work just with a subset of packages often used
in small systems - both Debian 'base' and such things as Busybox and
uclibc. This is in order to get something that
is genuinely useful whilst keeping the number of packages and
maintainers we are dealing with down to a reasonable number. Once the
principle is established we'll include more packages.

Each package will have a set of patches in emdebian SVN that
implement the changes necessary to make an emdebian archive.
In general, changes will be generated by scripts in the
emdebian-tools package with as little manual editing
as possible.

These new files will be used by the normal Debian build infrastructure
with assistance from wrapper scripts from emdebian-tools. Tools
will separate out translation files into generated packages - providing
users with the mechanism to only install translation files for the
languages in use. Tools will also remove documentation and (where possible)
package optional components separately.

For some packages it will be best to change options to reduce the number
of dependencies and/or be better suited to small systems.All packages
will be built with -Os. Some secondary binary packages probably need not
be built at all.

All this means that emdebian will have a different dependency tree to
Debian, and auto-building Debian is already tricky due to many circular
dependencies. Splitting out the documentation will remove many of these,
but managing dependencies such that Emdebian remains buildable at all
times will require care. A preliminary build of 550 packages with glibc
has shown that the problem is manageable. Ensuring that build-depends
are honoured for migration of a source package into Emdebian (unlike
Debian proper's migration policy from unstable to testing) should help;
so will keeping the total number of packages much smaller than Debian's.
Advice from release gurus is welcome.

We need to

Extend the build system for Crush (buildds)

Identify and maintain the subset of Debian packages
suitable for Emdebian Crush and for Emdebian Grip.

Make and update Emdebian patches for each package

Persuade maintainers to incorporate suitable Emdebian changes

Build away and see how well it works.

Document the process of creating an emdebian patch set for each
package

Document the build system so that people can monitor it and reproduce it

Develop emdebian package policy to ensure consistency and minimalism

Volunteers are needed for all these tasks, otherwise this scheme will not
get off the ground. Roll up, roll up.

Native compilation vs. cross-compilation

Debian is currently all natively compiled because some packages are
difficult to cross-compile. This is less true than it was, and most
of the things needed for Emdebian should cross-compile properly. For
many embedded developers cross-compilation is normal, and with
underpowered target platforms is often essential to get things
compiled in a reasonable time.

People using binaries pre-compiled by Emdebian don't care how they got
built so long as they work, but for those who need to compile their
own packages cross-compiling is usually necessary. Native
compilation works better for some architectures than others - e.g.
fast new ARM machines are available which make native ARM compilation
relatively quick. Native compilation means maintaining more build
machines and some of them will be slow, but it removes one source of
potential problems, and is compatible with the rest of Debian.

Recognising that most embedded developers need to be able to
cross-compile, the standard emdebian build system will use the stag
dpkg-based cross-compile environment. Thus a side-effect of the
project will be making much of Debian cross-compile properly. For
packages where this is not yet practical we will use scratchbox,
which lets compilation happen on fast machines with configure scripts
and other compile-time executables run natively.

Quality Checks

Cross building introduces a few, specialised, problems to do with
keeping the prebuilt packages installable and tracking issues in the
Debian to Emdebian progression of packages. The problems and issues
are described in the Quality Assurance
documentation for Emdebian.

Documentation

A detailed description of what
developers need to do is one of the tasks.

Emdebian is Debian, just smaller.

So, what is the relationship between Emdebian and Debian?

Emdebian is the distribution created by the Embedded Debian
Project, an official sub-project of Debian.

Emdebian and Embedded Debian is explicitly supported by core parts
of Debian, from ftp-master and release teams to gcc and glibc
maintainers. Debian has chosen to support making changes that assist
embedded development, via the Embedded Debian Project and has
consistently done so since 2000 (around the time of Potato, if not
before).

Emdebian can be completely regenerated using nothing but Debian
(except possibly during a release freeze in Debian where interim
releases exist outside Debian for the length of the freeze). In fact,
Emdebian relies on Debian to be able to build Emdebian itself - building
Emdebian on Debian derivatives or other GNU/Linux operating systems is
explicitly unsupported.

Emdebian consists mostly of Debian Developers and actively promotes
new members of Emdebian to become Debian Developers themselves.

Embedded Debian is determined to hold to all the Policies and
practices of Debian that are compatible with an embedded target and to
include such changes into Debian that allow for the number of actual
differences to be as small as possible.