## Using pbulk to create a pkgsrc binary repository
*pkgtools/pbulk* package consists in a set of tools designed to ease
mass-building of *pkgsrc* packages, and the creation your own *pkgsrc* binaries
repository.
Its setup needs a bit of work, here is how to prepare and run your bulk-build
box. In this article we will only consider a one-node machine.
This documentation is based on the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup).
### Prerequisites
These are the prerequisites needed by *pbulk*:
* A *pkgsrc* source tree
* Possibly a *src* source tree, only some rare packages needs it. If you choose not to checkout *src*, simply create a */usr/src* directory so *mksandbox* (see below) doesn't complain about non-existing directory.
* Possibly (not mandatory) a tool like *misc/screen* or *misc/tmux* as the full build process can take a **very** long time.
For example, to prepare a *pkgsrc-2011Q3* bulk build:
# cd /usr
# cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3 pkgsrc
Avoid automatic update of *pkgsrc* tree (cron or such), if you're in the middle of a build, it could lead to unfortunate results.
### Prepare a chroot
In order to isolate the bulk build, it is advised you run all the operations
within a *chroot*. Running *pbulk* on your real environment would wipe all
of your installed packages, and would modify your base system with lots of
directories, users and groups you don't need.
Fortunately, a tool called *mksandbox* will simplify this process. *mksandbox*
is located in *mk/bulk/mksandbox*, and it is called like this:
# mk/bulk/mksandbox [optionnal flags] /path/to/sandbox
For example, to create a sandbox in */home/bulk* without the X11 system, run:
# mk/bulk/mksandbox --without-x /home/bulk
This command will prepare and mount most of the needed directories, and will
place a shell script on top of the sandbox filesystem called *sandbox*. This
script is used to *mount*/*umount* your sandbox. It is a good idea to add
*/var/spool* to the list of directories mounted as read/write in your sandbox
so the email report is actually sent. Simply add:
/var/spool /var/spool rw \
to the list of directories in the *sandbox* script. *sandbox* script use is
really straightforward:
# /path/to/your/sandbox/sandbox umount
Unmounts the sandbox
# /path/to/your/sandbox/sandbox mount
Mounts the sandbox
### Prepare the pbulk environment
Now that our sandbox is available and mounted, we will chroot to it:
# chroot /home/bulk /bin/ksh
As the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup) suggests, we will create some helpers directories to keep things clean:
# mkdir /bulklog # bulk meta data and log files of each package
# mkdir /scratch # WRKOBJDIR
# mkdir /distfiles # DISTDIR
# mkdir /packages # PACKAGES
The */scratch* directory must be writable by a *bulk* user defined below in */usr/pkg_bulk/etc/pbulk.conf* as the build is done using DESTDIR.
# useradd bulk
# chown bulk /scratch
It is now time to bootstrap (yes, even on NetBSD) a *pbulk* environment so the latter is not wiped out when a new bulk build is started:
# cd /usr/pkgsrc/bootstrap
# ./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb
# rm -rf work
Add the line:
PACKAGES= /scratch/pbulk_packages
to *pbulk*'s prefix *mk.conf* (*/usr/pkg_bulk/etc/mk.conf*).
Then install *pbulk* itself on this newly created separate prefix:
# cd /usr/pkgsrc/pkgtools/pbulk
# env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake install
### Configuring pbulk
*pbulk* configuration file is */usr/pkg_bulk/etc/pbulk.conf*, here are
some parameters you should modify to customize the run to your needs:
base_url= # should point to the URL where the txt report will be
master_mode=no # this will not be a distributed build
# [...]
report_recipients="your@email.com"
# [...]
#bootstrapkit= must be commented on NetBSD !
unprivileged_user=bulk # used for unprivileged user-destdir builds
# [...]
bulklog=/bulklog
packages=/packages
prefix=/usr/pkg
pkgsrc=/usr/pkgsrc
pkgdb=/var/db/pkg
varbase=/var
# [...]
make=/usr/bin/make # must be modified on NetBSD !
Also, in order to avoid hangs, it might be a good idea to add the following to the top of *pbulk.conf*
ulimit -t 1800 # set the limit on CPU time (in seconds)
ulimit -v 2097152 # limits process address space
### Configuring the build
Last but not least, you should configure your */etc/mk.conf* (in the *chroot* !).
Here's an example of what we use at [NetBSDfr](http://www.netbsdfr.org/):
WRKOBJDIR= /scratch
PKGSRCDIR= /usr/pkgsrc
DISTDIR= /distfiles
PACKAGES= /packages
FAILOVER_FETCH= yes
SKIP_LICENSE_CHECK= yes
ALLOW_VULNERABLE_PACKAGES= yes
PKG_DEVELOPER?= yes
# site specific changes
PKG_OPTIONS.irssi= perl inet6
PKG_OPTIONS.mplayer= oss
DSPAM_STORAGE_DRIVER= mysql
PKG_OPTIONS.dspam+= graphs
PKG_OPTIONS.dovecot= ssl ldap dovecot-sieve dovecot-managesieve
PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers
X11_TYPE= modular
### Running the build
Now that everything's in place, we can fire up the build from the *chroot* using
the following command:
# /usr/pkg_bulk/bin/bulkbuild
It is recommended to run the build inside a tool like *misc/screen* or
*misc/tmux* as it will take a lot of time.
If the build is stopped, it is possible to restart it by invoking:
# /usr/pkg_bulk/bin/bulkbuild-restart
### Hints
If you'd like to restart the build to retry building some failing packages, **empty** */meta/error* (don't delete it):
# printf "" > /meta/error
If you'd like to rebuild a single package, use the *bulkbuild-rebuild* command followed by the package name.