Current and future work on sbuild (including wanna-build and buildd) - Debian

This is a discussion on Current and future work on sbuild (including wanna-build and buildd) - Debian ; Hi folks,
(CC'd to -devel for comments about the database stuff at the end, and
also the chroot-* symlinks if anyone still uses them.)
Just to let you know what I'm currently working on, and planning to work
on:
Completed ...

For a few years now, schroot has been used to access chroots, with the
older sudo access method deprecated, obsoleted and removed. However, a
few people expressed a need for sudo support in order to have behaviour
identical to buildd sbuilds, including non-functional networking and
having apt and dpkg run outside of the chroot environment. While I
didn't have time before the Lenny freeze, it's now got to the top of my
ad-hoc TODO list. This mostly involves re-merging already existing (but
removed) code back into the current codebase.

A split apt/dpkg chroot setup is not possible at the moment, so I've
been looking at adding back sudo support to sbuild. With the
object-orientation of sbuild over the last year, it's now possible to
cleanly abstract this out by deriving sudo- and schroot-specific
variants of the Sbuild::Chroot and Sbuild::ChrootInfo classes, special
cases being the handling of apt and dpkg runs.

The above URL shows the sudo-specific changes I've made over the last
day or so. It should be done and finished later on in the week, time
permitting.

Chroot symlinks:

schroot uses system wide chroot definitions in /etc/schroot/schroot.conf
and /etc/schroot/chroot.d. sbuild, conversely uses chroot-* symlinks
(or directories) in the current build directory. I was thinking of
requiring symlinks to the sudo-accessible chroots in /etc/sbuild/chroots
instead. Does anyone have any comments about that? Since sudo access
is required to chroot into them, is there any reason for them to be
present in the user build directory, especially now that ~/build is
entirely optional; it's now just $PWD.

2. Object orientation
----------------------

Other things recently done are the complete object-orientation of
option parsing and configuration handling. Option parsing now directly
wraps the configuration class, making it possible to drive the sbuild
Sbuild::Build and Sbuild::Chroot classes independently of the
command-line option parsing, making them embeddable in other code (I'm
thinking of buildd here, which will no longer need to invoke sbuild; it
just needs to instantiate and use the objects). buildd currently messes
inside the chroot prior to invoking sbuild, which can break schroot if
using it with session management (the chroot won't exist or be
accessible until after sbuild sets up the session).

This is now complete for sbuild and all sbuild-* helper programs. This
still needs doing for wanna-build and buildd.

Squeeze goals
=============

1. Refactoring of build code
----------------------------

Once these things are done, I'd like to tackle some larger problems for
Squeeze by refactoring the Sbuild::Build class. This would include
object orientation and cleanup of the dependency handling code, which
would allow us to fix most of the bugs in the BTS related to bad
handling of build-deps in corner cases, as well as allowing us to retain
backward compatibility with buildd sbuild (alternative build-dep
resolution algorithms are already selectable, though only used in a
single simple case).

2. Database usage
-----------------

I would also like to abstract out the handling of the build stats and
wanna-build databases to allow use of backends other than BDB and MLDBM,
since these are rather limited and slow. I'm thinking here of SQLite
and Postgres as alternative backends, while still providing support for
the old backends (if only a mechanism for dumping and upgrading to a
newer backend). It might also be useful to collaborate with the UDD
people if their database schema could accommodate what we need from it.
The wanna-build and stats databases should probably also be integrated
into the same database, since the stats can just be an extra table with
relations to a package table

Note: I did propose doing this a number of years back (2002), but this
was rejected due to adding too many extra dependencies to
sbuild/wanna-build/buildd, making bootstrapping new architectures too
costly. These would be strictly optional, so bootstrapping would still
be possible with a less functional build which just supported once
backend (equivalent to the current state).

3. wanna-build and buildd
-------------------------

wanna-build was refactored sufficiently to run for Lenny, but is still
in poor shape. buildd and all its helper programs exist in the source
tree, but are not yet tested and likely do not work as is. It would be
great to get these refactored in the same manner as sbuild was
refactored for Lenny (object-orientation, duplicated code elimination,
use of strict Perl). This also includes documentation, given that most
of the programs lack manual pages, and require a degree of arcane manual
configuration which has been almost entirely eliminated (and completely
documented) for sbuild.

In keeping with (2) above, I'd like the wanna-build database usage
completely abstracted out of all programs into a common module so that
all programs can easily interact with the database through a common
interface.