HowTo: Making a bootable ISO w/ miniroot.kmod etc.
I've asked how building a bootable ISO works now that we have kernel
modules and a bootloader that can load kernels before the kernel,
e.g. for a ramdisk. After some digging, I've managed to fit the pieces
together, get things going, and I've
posted a write-up to tech-install so others that
are intestested don't have to wade through that swamp again.

The posting describes how the build process was in pre-5.0/current
times (e.g. 4.0), what things change in 5.0/current now, and what
doesn't change.

Catching up - what happened in NetBSD-land between mid-August and mid-November
OK, I've been slacking^Wbusy for the past weeks, but I hope things
will get a bit better now. For a start, here's a catch-up of the
things that accumulated in my inbox in the past ~two months:

Google Summer of Code is over for some time, but apparantly
no final report has emerged so far (shame!). Still, a number
of individual status reports came by on the official lists:

I know of at least one other project (uvc) that has completed but
that I didn't see a report here - maybe I've missed it. Anyways,
GSoC was another big success this year. Thanks, Google!

Speaking of Adam Hamsik and Logical Volume Management (LVM), Adam
has continued his work in that are, and he has written a device
mapping library that interacts with his kernel driver. This
allows to interact with his GSoC project without using any GPL
code!
See Adam's posting for more details.

Force 10 Networks, producer of 10gbit switches that use an operating system
based on NetBSD, have added a new feature as part of their FTOS operating
system: VirtualView, which provides virtualization of Force 10 based equipment.
From the xchange article: ``Force10 Networks Inc. this week introduced VirtualView software for benchmaking, troubleshooting and managing virtualized environments based on Force10 gear.''

Following the latest hype in portable computers, NetBSD has
created a netbook
page that intends to list models and the extent to which they
are supported. Your contributions are most welcome here! (Contact
me for sending updates and hardware :-)

Zafer Aydogan has made RSS feeds available for CVS commits to
single files - see his mail to netbsd-users
for more details.

New security advisory were released that I've missed in my
last update:

A project that's been ongoing for quite some time is the move from
"old-school" loadable kernel modules (LKMs) to new-style kernel modules.
Important changes include the fact that modules can be either linked
into the kernel at build time, or loaded into the kernel at
runtime from the same file. Also, the bootloader was modified to
load modules after the kernel, e.g. for a RAM-disk like the one
that is used by the INSTALL kernel.

In the same line, some parts are starting to be moved out of the
GENERIC kernel, and installed as modules that can be loaded by
the new framework then. The start is made
by
POSIX semaphores as a first step and proof-of concept,
even if
some details are still under hot debate, e.g.
what the file system layout for modules is, and if the belong
to the kernel and its build process, or to the userland.

While talking about splitting the kernel into modules, Antti
Kantee has continued his work to move parts of the kernel into
userspace, in particular running file system code as userland in
his RUMP,
and puffs and (Re)FUSE works.
The idea is to provide the interfaces that file systems need in
the userland, and the result is that you can run code that used
to run inside the kernel in userland now.

NetBSD has shipped XFree in previous releases, and people who
wanted to use X.org had to install it from pkgsrc. That's all
fine, but to get a modern X, one had to compile things, as no
precompiled binary packages are made available for many
platforms. This is changing now, and NetBSD is getting X.org
integrated via a reachover infrastructure which is also enabled
for crosscompiling.

The "user interface" for this is still in flux, but after some
detour ("build.sh -V MKXORG=yes", without -x), "build.sh -x" now
builds whatever X is considered the default for the
platform. Some platforms already default to use X.org as X, and
more will come, as changes that were made to NetBSD's copy of
XFree are adopted to X.org.

As X.org is at Revision 7 now, it's installed in /usr/X11R7,
which will lead to a lot of interesting effects. pkgsrc is
already prepared for the new layout, but there are still many
minor details that will need adjusting to the new directory. If
you find one, post your patches to tech-x11.

Besides the GNU C compiler, there's the BSD-licensed Portable C Compiler
around for some time now. It doesn't offer the same support as
its GNU cousin yet, but this may change now:
The BSD Fund
is currently doing a fund drive to get money to enhance PCC.
The goal is to raise $12,000US to improve support for core
compiler functionality as well as support for C99, gcc
compatibility and the amd64 architecture. See
the project page for further details.

As the final point today, a word on NUMA support from Christoph
Egger. Non-Uniform Memory Access is needed in massive parallel
systems where some nodes have RAM more tightly associated than
others, where the RAM is further away, resulting in different
access times for different regions of memory. In order to support
this, Christop Egger has made first steps.

His example implementation uses information from ACPI, and shows
some heavy dmesg-pr0n from a 16-core machine with four
sockets. Yumm!

So much for today. With the NetBSD 5.0 release cycle started, I'd like
to encourage everyone to test the release branch, report errors, send
patches as well as beer and choccolate to make this the best
release that we've ever had.

More kernel works: preemption and realtime, devfs, modules, testing
The following kernel-related projects were raised in the
past few weeks:

Kernel Preemption:
Andrew Doran has continued his work towards fine-grained
locking, and he has proposed a
patch to implement kernel preemption,
i.e. that in a realtime environment, high-priority processes can
interrupt system calls running inside the kernel.

Handling the Floating Point Unit (FPU) was
added later on --
the FPU needs special attention as saving and restoring is
expensive, and doesn't need to be done in many cases. But if a
program uses it, care must be taken to handle the case.
The exact handling is
explained by Christoph Egger.

While there,
Christoph also outlined the
roadmap for getting realtime support
in NetBSD - there are still a number of bits missing, but being
able to preempt the kernel is a good first step!

Fine-grained socket locking:
In order to allow fine-grained locking (instead of blocking
all other processes from entering the kernel, as is done in the
"biglock" SMP approach), many kernel subsystems need to be changed.
The socket system is the core part of interprocess communication,
and Andrew Doran has changed it to
use fine-grained locking
now.

In that context, the question of
what code still runs with the biglock held, and
Andrew gave an overview where
more work is needed: some file systems (lfs, ext2fs, nfs),
most of the drivers, protocols like TCP/IP, Veriexec, and
some machine-dependent parts.

Kernel modules and ramdisk:
A change in kernel modules was proposed
some time ago, and
Andrew Doran has used this scheme now to unify the way
many ports handle the install media: There, the kernel loaded
contains a ramdisk (miniroot) image inside the kernel, which is then used
as root-filesystem for the kernel, containing the install tools.

In order to split things and eventually use a stock GENERIC kernel
for both running and installing, Andy has
changed the
x86 boot process to load the miniroot as a kernel module.

Device File System (devfs):
Another area of the kernel where a lot of work is currently being
done by Matt Fleming is NetBSD's device driver infrastructure,
esp. under aspects of dynamic attaching, detaching, and suspending
(power management!). To talk to the various drivers, device nodes in
the /dev directory are kept right now, but those are static and
need to be updated when a new driver is added.
Matt is working on a Device Filesystem (devfs) that dynamically
created /dev from the list of devices inside the kernel. The
fileysstem will also handle dynamic creation and deletion of
nodes, and as an important case it will also keep permissions
across reboots, if someone changes permissions manually.