NetBSD Documentation: Porting NetBSD to a new ARM SoC

Written and contributed by Jonathan Sevy

Introduction

This article describes experiences in porting NetBSD to a new
ARM-based SoC. The specific chip addressed is the ARM9-based Agere Vx115
cellular telephony digital baseband chip, and the Vx115 VEP evaluation
board hosting the chip. The information presented here is intended to
assist with the porting of NetBSD to any new ARM-based platform.

Porting Variants

There are a couple of variants possible when porting NetBSD
depending on the level of support already available in the
kernel.

Processor architecture port

This involves adapting the NetBSD kernel to run on a
new processor architecture type (like MIPS, PowerPC, ARM, etc., though
those types are already supported). This is a major undertaking that
requires deep understanding of both the NetBSD core and the processor
architecture.

SoC port

This involves porting NetBSD to a new SoC which uses
as its processor core a type already supported by NetBSD but provides
some unique set of on-chip peripheral devices. This involves defining
the bus structures needed to interconnect the devices on the SoC, as
well as drivers for the peripherals.

Platform port

This involves developing code to allow NetBSD to run
on a particular platform (evaluation board) that uses a chip (SoC)
already supported by the NetBSD kernel. This involves creating
appropriate platform startup code, as well as support for off-chip
peripherals on the board, as well as an appropriate root filesystem with
the platform applications and libraries. This document describes the
latter two activities. The Vx115 SoC is based on an ARM9 processor core
already supported by NetBSD (though the exact variant, the ARM926EJ-S,
was not directly supported in the kernel version used and required minor
adaptation), so a processor port was not needed, but the SoC provides a
uniques set of on-chip peripherals, and was provided on an evaluation
board with various supporting hardware components.

Scope

The focus of this porting effort was to get NetBSD running on the
platform with only the minimum support needed to host a console and
shell on the serial port. Only those on- and off-chip peripherals needed
to support this operation were addressed; the SoC provides many other
peripherals which would need support (drivers) to comprise anything near
a full board-support package.

Such a "basic" port requires addressing a relatively small number
of issues, described in the following sections.

Each of these involves creating various configuration,
source and header files, and the contents of these files is discussed in
the subsequent sections.

Acknowledgments and References

As is standard practice with open-source systems, much of the
required functionality can be found in corresponding files for existing
ports. This project makes full use of this available source code (with
thanks to all of the original authors). In fact, it's because of the
work of others, and their openness in sharing it, that NetBSD and other
open-source systems are quite straightforward to adapt to new
platforms.

Building

I do most of my development work on an x86 Linux system, and thus
wanted to be able to cross-build the NetBSD system from within the Linux
environment. Fortunately, NetBSD provides a great build system that
supports both cross-compilation (building for a processor architecture
different from that of the build host) and cross-platform builds (building
the system on a non-NetBSD build host).

The build process consists of several steps:

Building the toolchain

Building the kernel

Building the userspace libraries and applications

Building the root filesystem

NetBSD comes with a wonderful shell script, build.sh,
that makes all of this easy.

Building the Toolchain

Building the toolchain can be accomplished using the build.sh
script: just run

./build.sh -u -m evbarm tools

from within
the top-level source directory.

Parameters:

-u: don't run "make clean" before
building; this prevents everything from being rebuilt if changes
are made (not really essential for this first build step)

-m evbarm: the machine architecture type;
this is for ARM evaluation boards

tools: build just the toolchain, not the whole
system

I actually found I had a problem when trying to build
the toolchain on a Fedora Core 5 system because the native Fedora tools
(which are used to build the NetBSD toolchain) were too new (gcc 4.1).
The NetBSD toolchain would build cleanly only with gcc version 3 tools.
Luckily, Fedora provides optional packages for the GCC 3.2 toolset,
which can be installed with

to ensure the gcc-3-based tools
are used to build the toolchain. Note that these tools (and the export
statements) aren't needed once the NetBSD toolchain is built - they're
just needed to "bootstrap" the toolchain itself. Note also that the
latest NetBSD distribution should build cleanly with the gcc 4-based
tools, so this step shouldn't be necessary.

Building the Kernel

Building the NetBSD kernel is also accomplished using the build.sh
script: just run

./build.sh -u -m evbarm kernel=VX115_VEP

from
within the top-level source directory.

Parameters:

-u: don't run "make clean" before
building; this prevents the toolchain from being rebuilt

-m evbarm: the machine architecture type;
this is for ARM evaluation boards

kernel=VX115_VEP: the kernel configuration
file to use; in this case, it's for the VX115_VEP verification
board. The configuration file is in sys/srch/evbarm/conf.

This command performs a number of operations:

creates a build directory
sys/arch/evbarm/compile/obj/VX115_VEP

runs the nbconfig "config" utility on the VX115_VEP
configuration file to create required source files for the build.
This config command does quite a lot; it actually creates
source files for the autoconfig framework specific to the system
and device configuration. This is discussed in more detail in the
section on
autoconfiguration.

runs "make depend" on the source files to create the
dependency tree for the source files

runs "make" to build the kernel and drivers

When the command completes, it will create the kernel
objects in the sys/arch/evbarm/compile/obj/VX115_VEP directory. The
object files will include the ELF kernel object, and may contain
the output in different formats depending on the configuration.

Building the Userspace Libraries and Applications

Since the NetBSD distribution includes all of the userspace
libraries and applications, the build.sh script handles these, too: just
run

./build.sh -u -m evbarm

from within the top-level
source directory. This will create a directory obj/destdir.evbarm with
all of the userspace files.

Building the Root Filesystem

The default build for NetBSD creates a filesystem for a desktop
system, which is way too big for an embedded system. I pulled out just
the minimum basic libraries and apps needed to have a console into a
separate directory called rootfs_ramdisk. (In fact, this is almost
certainly not the bare minimum, but was small enough to fit in the
ramdisk, so I didn't bother to strip it further.)

This directory is a directory of files on the build host
filesystem (Linux, in my case, using an ext3 filesystem). It's necessary
to create a single file which contains the filesystem image for NetBSD -
that is, the set of objects which are currently in the build host
filesystem, but packaged such that they form a NetBSD filesystem. This
is done with the tool nbmakefs, which is built as part of the
toolchain.

nbmakefs -s 5000k rootfs_ramdisk.img rootfs_ramdisk

This will create a filesystem of total size 5000 kB - the resulting
image will be 5000 kB in size (or about 4.9 MB). This value is also used
in the kernel configuration to "reserve" this much space in the kernel
for the filesystem ramdisk image.

Inserting the Ramdisk Image into the Kernel

The ramdisk created above is inserted into the kernel using the
tool mdsetimage:

arm--netbsdelf-mdsetimage -v -s netbsd rootfs_ramdisk.img

where netbsd is the kernel image created from the kernel build
process.

Further Packaging

For the evaluation platform here, the platform loader used
s-record format for the loadable files, so it was necessary to use the
objcopy tool to generate the loadable srec from the kernel+ramdisk
image. In addition, the kernel was to be loaded at address 0x20400000 in
the system Flash, while it was configured to execute at 0x24300000,
which required the --change-addresses argument during s-record
generation. This sort of thin is very platform-specific, so these steps
may not be relevant for another platform.

Device Autoconfiguration Process

The NetBSD device architecture is a tree consisting of children
(devices and buses) descended from parents (buses). The "family tree" of
devices is (partially) defined in kernel configuration files named
files.*, where each child (device or bus) is specified as being attached
to a specific parent (bus). The syntax of this attachment is discussed in
the kernel configuration section ,
and described in detail in the NetBSD
man page on config . The root of the tree is the "root" bus;
all other buses and devices are descendants of this. In particular, for
ARM-based systems, the bus "mainbus" is attached to the root in the file
files.arm, and this is the bus to which the cpu is attached (in files.arm)
as well as all system buses (in the processor-specific
files.vx115).

The above defines the basic relationship between devices and buses.
However, the actual instances of a specific device or bus on a platform
are defined in the main platform configuration file - here, the file
VX115_VEP. In this file, instances of buses and devices are specified with
the bus/device base name followed by a digit - for example, mainbus0.
There may be multiple instances of the same device, differentiated by
their numerical suffixes - for example, vx115_com0 and vx115_com1. In
addition, the specification of a device instance can provide a device
locator, which is a list of parameters specific to that instance - for
example, the base address of the device instance, IRQ, etc. The
parameters that make up the locator, along with default values, are part
of the parent (bus) specification in the files.* files. These parameters
are used during the autoconfiguration process, described below.

The tree structure of devices is built by the kernel configuration
process, in which the NetBSD config utility is run on the platform
configuration files to create source files with data structures reflecting
the specified device relationships. The tree structure created by the
kernel configuration process is used during kernel initialization in a
process called autoconfiguration. This involves the sequential
initialization of devices in a depth-first search of the tree of devices,
as follows.

Each bus and device defines two methods, match( ) and attach(
), and registers them with the autoconfiguration system using the
CFATTACH_DECL macro.

During kernel initialization, each parent (bus) instance first
probes its child instances by calling their match( ) functions,
supplying the information (such as the device address and other
parameters) that was specified in the child device's instance
specification. The match( ) function uses the supplied parameters to
determine if the information corresponds to that child, and if so,
returns a non-zero value.

When a child instance has returned that it's a match, the
parent then calls the device's attach( ) method, in which the child
instance initializes itself. If the child is itself a parent (bus),
it will probe its children as part of its attach( ) method.

Each parent uses the NetBSD config_search method
and its own search method to probe its children. config_search provides
the parent search method with a cfdata struct which contains the device
instance-specific locator parameters defined in the device instance
specification. This data is passed to the match( ) and attach( )
functions. The cfdata structure associated with a device has a very
general structure, with a loc field that's an array containing the
bus-specific locator parameters with array indices that are defines
generated by the config utility from the parameter names in the bus
specification. For convenience, the parameters are usually copied into a
more "structured" struct before being passed to match( ) and attach(
).

A Few More Details

The parent device search routine uses the NetBSD kernel function
config_match function to look up and call its children's match( )
functions. The config_match( ) function calls the match( ) function for
a device as follows.

config_match( )

calls config_cfattach_lookup( ) with device name as
argument

calls config_cfdriver_lookup( )

runs through allcfdrivers list

returns cfdriver with matching name

returns config_cfattach_lookup_cd( ) with driver as
arg

runs through cfdriver's cfattachlist field to
find name match

returns cfattach which matches name

call the cfattach struct's field cf_match = match function
of child

The cfdata structures passed to parents (buses) during the
autoconfiguration process are generated in the file ioconf.c during the
configuration process from files.vx115 and VX115_VEP. Each bus structure
(parent) defines cfdata locator fields in files.vx115; the format of the
declaration of a bus and its locator fields is

device foobus {[field1 = default1], ... [fieldn = defaultn]}

field1 through fieldn are locator names; the configuration
process generates defines from the locator names with names
FOOBUSCF_FIELD1, ..., FOOBUSCF_FIELDN, which are used as the array
offsets to the corresponding information in the cfdata struct's
loc array

This bus thus defines 4 locators, which result in the generation of the
defines VX115_APBCF_ADDR, VX115_APBCF_SIZE, VX115_APBCF_INTR, and
VX115_APBCF_INDEX for the locator positions in the cfdata struct's loc
array, and the default-value defines VX115_APBCF_ADDR_DEFAULT = -1,
VX115_APBCF_SIZE_DEFAULT = 0, VX115_APBCF_INTR_DEFAULT = -1, and
VX115_APBCF_INDEX_DEFAULT = 0

Device instances and their locator values for cfdata loc fields
are specified in VX115_VEP. The format of the device instance
specification is defined in the NetBSD man page config(5).
The declarations for the timer device attached to the Vx115's APB
is

vx115_clk0 at vx115_apb? addr 0x700C5000 size 0x68 intr 9

Any field not specified gets default value in cfdata loc field, so in
this case index = 0. To make it easier to work with the locator values
outside of the rather unstructured cfdata loc array, it's typical to
define a struct for the bus that mirrors the cfdata loc fields. For the
Vx115's buses we define in vx115_var.h the struct type

This is initialized when the bus's devices are being probed from
the corresponding device's cfdata struct, and then passed to the
device's match( ) and attach( ) functions; note the use of the locator
defines as offsets in the loc array of the cfdata struct.

Kernel Configuration

As discussed in the previous section, there are a number of files
that specify the configuration for the NetBSD kernel. Some of the files
are specific to the processor, and shared by any machine which uses that
processor, while others are specific to the machine (development
board).

Directories

The files (configuration, source and header) associated with an
ARM SoC or platform port are placed in subdirectories created for that
SoC or platform. The SoC-specific directory is created in the
sys/arch/arm directory; it's sys/arch/arm/vx115 in this case. The
platform-specific directory is created in the sys/arch/evbarm
directory; here, it's sys/arch/evbarm/vx115_vep.

Processor Configuration Files

These are files that are specific to the SoC, and present
regardless of the board it is used on. These files are placed within the
sys/arch/arm/<processor-name> subdirectory (sys/arch/arm/vx115 in
this case).

This specification file contains fundamental device
specifications, attachment indications and source-file inclusions for
peripherals associated with the processor, i.e., used by any platform
based on the processor. This includes things like

basic irq and bus operations

system buses (we have an AHB and APB in this case)

system timer (clock) used for the fundamental system time
reference

interrupt controller

console serial interface (UART)

This file is used by the NetBSD autoconfiguration
system, to include appropriate source files for compilation and to
create appropriate structures reflecting the organization of the
system (which peripherals are attached to which bus, what parameters
the buses pass to their children, etc.) This file is included by the
board-specific configuration file (in this case, files.vx115_vep , described
below).

As can be seen in files.vx115 , each bus or device has 3
basic parts to its specification

device <device-name>

Provides a name for the device, for use in further
references in the code and configuration
files.

For buses, this also supplies a list of the
parameters that it will supply to its children when they
are initialized by the bus, as well as default values for
the parameters if specific values are not specified for
the child in the machine-specific configuration file (here
VX115_VEP , discussed
below)

attach <device-name> at
<bus-name>

Indicates the device is to be a child of a
particular bus; this is used during autoconfiguration,
when each bus probes for and initializes its
children

file <source-file-path>

Indicates the specified file should be included in
the compilation

The vx115.files configuration file has just the
basic set of configuration options (buses, interrupts, timers and
serial interface) needed to initially bring up the system. Additional
lines would be added as support is added for each of the additional
on-chip system peripherals.

Note that the system AHB and APB buses are attached to something
called "mainbus". This is the root bus in NetBSD. The CPU itself (the
ARM926 core) is attached to this as part of the standard ARM system
configuration (specified in the file
sys/arch/arm/conf/files.arm).

Board Configuration Files

These files provide configuration for features that are
board-specific. Note that this includes things like startup code (since
this depends on the specific memory selects used on a board), as
well as the exact specification of kernel features to be
supported.

This is the main config file, passed as argument to
nbconfig. It's similar to the Linux .config file.

Include std.vx115_vep (which includes files.vx115_vep and
specifies mk.vx115_vep)

All other main kernel options

Specifies instances of devices and buses defined in
VX115_VEP, and locator parameter values to be passed to devices
during autoconfiguration. The syntax of this device instance
specification is described in the autoconfiguration section, and in
detail in the NetBSD
config(5) man page .

# The main bus device

mainbus0 at root

# The boot cpu

cpu0 at mainbus?

# Vx115 AHB and APB

vx115_ahb0 at mainbus?

vx115_apb0 at mainbus?

# On-chip interrupt controller

vx115_pic0 at vx115_apb? addr 0x700C1000 size 0x14c

# On-chip timer

vx115_clk0 at vx115_apb? addr 0x700C5000 size 0x68 intr
9

# On-chip serial UART

vx115_com0 at vx115_apb? addr 0x700E2000 size 0x7c intr
19

Initialization

Initial startup

The first code executed in the NetBSD kernel is low-level startup
code provided as part of the board-specific software. This code is
specified by the SYSTEM_FIRST_OBJ and SYSTEM_FIRST_SFILE defines in the
Makefile fragment mk.vx115_vep
(discussed in the section on
configuration ); the source file is vx115_vep_start.S in this
project.

This assembly file is responsible for setting up the system so
initial kernel execution can take place. It performs the following
operations.

Copy the kernel to RAM

Set up initial MMU page tables, using all section (1 MB)
maps to start. In particular:

Map all physical addresses to same virtual addresses
(VA == PA), for any peripherals that may need this

Do this so can use bus mapping functions during
setup in the init_arm( ) function

Map the RAM with VA == PA, but set to be
cacheable

Map kernel virtual addresses to RAM physical
addresses

Enable the MMU

The initial MMU mapping is really just to allow the
kernel to start executing from its relocated location in RAM, before it
sets up its real page tables in the init_arm( ) routine, and to allow it
to access peripherals during configuration.

Machine startup

The file vx115_vep_machdep.c provides
platform-specific initialization. This file defines several functions
and data structures: an initarm( ) function that performs basic system
initialization, a cpu_reboot( ) function that restarts the system on
reboot, and a pmap_devmap structure that defines the peripheral
virtual-to-physical memory mapping. Fortunately, the vast majority of
these functions are common across platforms, so existing code can be
used with minor modifications. The vx115_vep_machdep.c file was based on
the smdk2800_machdep.c source, and used the cpu_reboot function verbatim
and the initarm function with minor changes.

struct pmap_devmap vx115_vep_devmap[ ]

array of specifications of virtual-physical memory mappings
for peripherals

initarm( )

The initarm( ) function provides basic machine setup. It's
called during initialization from arch/arm/arm32/locore.S, before the
kernel main( ) function is called in kern/init_main.c. The
initialization performed in this function is as follows.

call set_cpufuncs to set up basic cpu functions

call pmap_devmap_register(vx115_vep_devmap) so can use
bus_space_map; the appropriate ranges must have been mapped
in vx115_vep_start.S

call vx115_intr_bootstrap(PIC1_BASE_PHYS, PIC1_SIZE) to do
early setup of interrupts needed for spl functionality

call pmap_bootstrap to initialize the BSD page map
structure from the ARM-specific page tables built above

return stack pointer

consinit( )

The consinit( ) function is used to initialize the serial
console just sufficiently to allow console messages to be printed.
Further initialization of the serial interface is deferred to the
autoconfig process.

Bus Operations

A NetBSD system defines bus operations that are used to perform
various memory operations. Each bus type specifies its set of operations
in a bus_space struct generally referred to as a bus tag. This struct
specifies many types of memory operations; NetBSD defines many
generic operations, and any bus-specific operations can be defined
as needed. The vx115_io.c file defines a
bus_space struct called vx115_bs_tag, as well as several bus-specific
operations used in this struct.

Each bus instance selects in its attach( ) function an appropriate
bus tag that defines the operations for that bus type. The bus then passes
the bus tag to its children during autoconfiguration. The children use
this tag in calls to the standard NetBSD memory functions such as
bus_space_map( ), bus_space_read_4( ), etc.

Interrupt Handling

Interrupt handling is part of the processor-specific port, since the
interrupt controller is part of the Vx115 SoC. The interrupt handling is
contained in two files, vx115_intr.h and vx115_intr.c , in the arch/arm/vx115
directory. Writing the interrupt handling code has two primary components:
developing code to handle system priority level (spl) changes, and writing
the actual interrupt handler. The SPL management is here because it
involves enabling and disabling subsets of interrupts.

System Priority Level (SPL) Handlers

The SPL handlers are used by the system to change the system
priority level (no surprise...), which amounts to masking interrupts
corresponding to lower-priority sources. This is the principal mechanism
that NetBSD uses to synchronize access to critical data structures in
drivers by code running in thread context and code running in interrupt
context: by preventing interrupts, the code is protected from access by
another task (preemption is disabled with interrupts disabled) and from
access by an interrupt handler (if the relevant interrupt has priority
at or below the current level, which is the natural situation). It also
protects timing-sensitive code against unexpected delays due to
background (lower-priority) interrupt activity.

A block of code is protected by enclosing it between calls
to spl<level>( ) and splx( ), where there's a separate
spl<level> function for each of the available system levels -
splserial( ), splclock( ), splnet( ), etc. The complete list can be
found on the spl man page.

For example, to protect against network-related interrupts and
those at lower priority, code like the following would be used:

int oldSPL = splnet(newSPL);
/* protected code here */
splx(oldCode);

This functions somewhat like the Linux functions local_irq_save( )
/ local_irq_restore( ) and disable_irq( ) / enable_irq( ). It
gives finer-grained control over which interrupts are disabled than
local_irq_save( ), which just disables all interrupts, but provides more
timing control than disable_irq, which disables just a single interrupt
and thus doesn't suppress interrupts at lower priority.

The various programmer-accessible spl functions are implemented in
terms of several low-level functions that are implemented for the
platform.

int _splraise(int);
int _spllower(int);
void splx(int);

These functions do what you'd expect: they raise,
lower, or set absolutely the system priority level by masking
lower-priority interrupts and leaving those above enabled. The raise and
lower functions are very similar to the splx( ) function, but first
check to see if a change needs to be made. The splx( ) and
_spllower( ) functions also check to see if any soft interrupts have
been unmasked, as discussed below.)

There are really only two issues for the spl functions:

how to mask interrupts

which interrupts to mask for a specified priority
level

The first issue is hardware dependent, and usually
quite straightforward: the system interrupt controller usually has a
mask register that allows interrupts to be masked or unmasked - writing
a '1' in a bit enables the interrupt corresponding to that bit, while a
'0' disables the interrupt, or some such approach. The second issue -
determining which interrupts to mask at a specific system priority level
- is only a bit more involved, but leads to some somewhat ugly
code.

For the ARM architecture, most platforms specify the interrupts to
enable at each system priority level using an array that maps the system
priority level to the specific interrupt mask to be used at that
level.

int vx115_pic_spl_mask[NIPL];
int vx115_pic_spl_soft_mask[NIPL];

Here, NIPL is the number
of system priority levels (defined in arch/evbarm/intr.h). The
vx115_pic_spl_mask array then holds the mask to use for the interrupt
controller corresponding to each system priority level. When the
priority level is changed, the mask in the array corresponding to the
new level is written to the interrupt controller:

(The Vx115 interrupt controller has separate registers for
enabling and disabling interrupts, hence the two write
operations.)

The second array, vx115_pic_spl_soft_mask, is needed because in
addition to masking hardware interrupts, the spl functions must also
manage "soft" interrupts. Here, a soft interrupt isn't the same thing as
an ARM software interrupt (the SWI instruction), but refers to a
lower-priority routine used in conjunction with a higher-priority
hardware interrupt handler. NetBSD soft interrupts are provided so a
high-priority interrupt service routine can defer some lower-priority
processing to avoid holding the system at a high priority level for an
extended period. The device initialization code calls
softintr_establish( ) to register the soft interrupt routine at one of
four soft interrupt priority levels, and then the hardware interrupt
handler calls softintr_schedule( ) when it wants the routine executed.
Soft interrupts are treated just like hardware interrupts, being called
from within the platform's interrupt dispatch code (discussed below),
but after all pending hardware interrupts have been
handled.

Just as with hardware interrupts, soft interrupts are associated
with a priority level, but the soft interrupt priorities are lower than
any of the hardware interrupt priorities. Thus when the system is at a
specific priority level, all soft interrupts at or below that level must
be blocked. The vx115_pic_spl_soft_mask array values specify which soft
interrupts should be enabled at each system priority level. The hardware
interrupt dispatcher (discussed in more detail below) tests to see if
any software interrupts are waiting to run, and executes them if any are
currently enabled, i.e., unmasked according to the value in
the vx115_pic_spl_soft_mask array corresponding to the current
system level:

Probably the best way to view the soft interrupt masks is as
extensions of the hardware interrupt masks: it's as if the mask register
has 64 bits instead of just 32 bits, with the lower 32 corresponding to
hardware interrupts and the upper 32 corresponding to software
interrupts (though only four bits are used at present). Some other
evbarm platforms utilize unused bits within the hardware mask to
represent software interrupts, avoiding the need for "extension" of the
mask field. However, for the Vx115, all 32 bits in the hardware mask
correspond to existing hardware interrupt sources, so
the "extension" masks were needed. This also helps to logically
separate the hardware interrupts from the software interrupts (but
complicates the process of initializing the masks).

How the arrays are initialized and updated is the messy part. I
used code for vx115_pic_init_interrupt_masks( ) and
vx115_update_intr_masks( ) that has seemingly been propagated from
system to system, and modified it to handle initialization of the dual
arrays. The process of array initialization and modification is the
following. Note that a '1' bit in a mask indicates that the interrupt
corresponding to the bit is to be enabled, and a '0' indicates it is to
be disabled.

vx115_pic_init_interrupt_masks( )

called at system init time (from the interrupt controller
attach function)

sets up the initial soft interrupt hierarchy; this affects
only the soft-interrupt mask array, as soft interrupts are the
lowest priority in the system and masking at a soft interrupt
priority level won't affect hardware interrupts (at least not on
our system)

vx115_update_intr_masks( )

called whenever a new hardware interrupt is registered, with
the irq number and associated priority level as arguments

select appropriate array for modification: if the irq number
is below 32, it's a hardware interrupt, so the hardware interrupt
array is selected, while if the irq number is 32 or above, it's a
software interrupt and the software interrupt is selected

clear the bit corresponding to the supplied irq
number in all the masks in the array for the specified
priority and below, and set the bit in the masks for higher
priorities; in this way the interrupt will be disabled at this and
higher system priority levels, and enabled at lower system
priority levels

do some "consistency magic": make sure that the settings at
certain priority levels include or exclude the settings at other
levels. This was copied verbatim from another platforms, and
should probably be revisited...

reset the interrupt controller mask register to reflect the
change

Interrupt Registration and Dispatching

Interrupt handlers are registered (usually in device attach
routines) using vx115_intr_establish( ). The platform interrupt
subsystem maintains an array of the handlers registered for each irq
number. The Vx115 platform currently allows only one handler per irq;
however, there's nothing preventing the use of a linked list per irq to
allow the registration of multiple handlers per irq, if desired. Each
handler is registered along with a "cookie", an arbitrary void* argument
that's supplied to it when it's run, and the priority level for the
interrupt. The vx115_intr_establish( ) routine calls
vx115_update_intr_masks( ), which sets the system priority level masks
to take into account the new irq at the specified priority level.

vx115_intr_establish( )

The main interrupt dispatcher, vx115_irq_dispatcher( ), is
called from the low-level IRQ vector assembly routine to dispatch
interrupts to the interrupt handlers corresponding to the currently
asserted interrupts. The dispatcher has a simple structure.

vx115_irq_dispatcher( )

save current spl (note that interrupts are enabled - not
globally disabled - when the interrupt handler is called)

while interrupt status register is nonzero

select one of the asserted irqs

raise the spl to the level corresponding to the
selected irq

call the handler registered for that irq

clear the interrupt in the controller

restore the saved spl

if any soft interrupts are asserted, dispatch to associated
soft interrupt handlers

Soft Interrupt Registration and Dispatching

Soft interrupts are registered through a call to the generic ARM
soft interrupt routine softintr_establish( ), which manages the list of
soft interrupts. Note that the generic soft interrupt handling code
specifically supports the registration of multiple handlers for each
soft interrupt level. The platform-specific code therefore doesn't play
a role in soft-interrupt registration.

Soft interrupts are handled much like hard interrupts: when a soft
interrupt is pending and the system priority level is lower than the
priority of the interrupt, any associated handlers are executed. The
platform maintains a static bitmask named softint_pending to determine
which soft interrupts are currently asserted, just like the hardware
interrupt status register. Of course, since this is just a software
variable, there's no hardware mechanism for informing the system when
the soft interrupt status has changed, i.e., when a new bit has been set
in the softint_pending variable. The system thus must test the variable
at appropriate times to determine if there's a soft interrupt
waiting:

at the end of the hardware dispatcher, since a soft
interrupt may have been scheduled by one of the hardware interrupt
handlers

when the system priority is lowered, since there may be a
soft interrupt that has been "masked" because the system priority
level was too high for it to run

when a new soft interrupt is scheduled

In each of these places, there's a call to
vx115_do_pending( ) to execute any asserted unmasked soft
interrupt(s).

_setsoftintr( )

A soft interrupt is scheduled for execution through a call
to the generic ARM soft interrupt routine softintr_schedule( ), and
this calls the platform-specific routine _setsoftintr( ). For the
vx115, this routine is quite simple:

set the appropriate bit in a static variable indicating
which soft interrupts are asserted

if the system priority level is lower than the priority of
the associated soft interrupt, call vx115_do_pending( ) to
execute the asserted soft interrupt(s)

vx115_do_pending( )

The vx115_do_pending( ) routine is called to execute
any pending soft interrupts. It's called at specific times, as
discussed above. The routine is simple:

while softint_pending masked by the
current vx115_pic_spl_soft_mask is nonzero (has bits
set)

check each of the four available soft interrupts; if
it's asserted and its priority is above the system
priority level

Initialization

The interrupt controller is really just a peripheral on the
Vx115's peripheral bus (APB), so it's treated like any other peripheral
as far as initialization is concerned. The interrupt controller
registers a match and attach function with the autoconfig system using
the standard CFATTACH_DECL macro.

However, the SPL services are called early in the boot
process, by the debug printf statements, before the autoconfig stuff is
run. Even though the SPL functions just set the interrupt mask, this
involves a register write, which uses the bus-write functions associated
with the interrupt controller's parent bus, which isn't assigned until
the controller's attach function is called. So there's an
early-initialization function, vx115_intr_bootstrap( ), that's
called to initialize just the bus-operations field in the softc struct
so the SPL functions will work (even though they'll have no effect since
the interrupt controller itself isn't initialized). Note that other
platforms just bypass the whole autoconfig process and "hardwire" the
interrupt controller.

System Timer

The system timer source code is in the source file arch/arm/vx115_clk.c , and provides the basic
NetBSD system time reference. The main functionality is as a driver for
one of the hardware timers on the SoC that responds to timer interrupts by
calling appropriate system functions to update the time. The timer source
also provides several other functions for services such as delays.

Initialization

System clock

The system clock provides ticks at the rate defined by the
configuration define HZ. The default value for HZ is 100, giving a
system tick period of 10 ms. However, because the Vx115 timer uses a 32
kHz clock as a source, for this platform HZ is defined as 64 to give an
integer value for the hardware timer max count value.

The system clock is started through a call to cpu_initclocks( )
from within the initclocks( ) function in kern/kern_clocks.c. This
starts the process of timer interrupt generation.

cpu_initclocks( )

register the timer interrupt handler

configure the hardware timer to start generating interrupts
at the rate HZ

vx115_clk_intr( )

handler for timer interrupts

call hardclock( ) so system can update time

delay( )

loop until at least the specified number of microseconds has
elapsed

microtime( )

return the time to the nearest microsecond

Time-of-day functions

resettodr( )

Serial Driver

The serial driver provides the console interface that's used to
deliver debug messages and interact with the system. The driver is
contained in the source file vx115_com.c , in the arch/arm/vx115
subdirectory. The driver provides several interfaces for use by various
NetBSD subsystems: the standard autoconfig functions, a set of character
device functions, and functions supporting console functionality.

tty functions

These functions are assigned to fields in the serial driver's
softc structure's tty field. The tty structure is allocated during the
attach routine, and then partially initialized by assigning these
function pointers. The NetBSD tty subsystem handles further
initialization (it obtains a pointer to the tty struct using the
vx115_com_tty( ) method of the cdevsw struct). These functions handle
the device-specific tty operations.

vx115_com_start

used by the tty subsystem to start output

assigned by the NetBSD tty subsystem during
initialization to be the t_linesw->l_start function
of the tty struct, which is called to transmit from
the tty buffer

get output parameters for the softc struct (the byte
count and output position in the tty output queue), for use
by the interrupt routines

mark the driver as busy so the softc parameters won't
be changed until transmission is done

turn on transmit interrupts so output will
begin

actual output is performed from within interrupt
routines

vx115_com_hwiflow

used to provide HW flow control; not used on this
platform

vx115_com_param

used for termios parameter changes (baud rate,
etc.)

cdevsw functions

These are the basic interface functions for character drivers in
NetBSD.

vx115_com_open

vx115_com_close

vx115_com_read

vx115_com_write

vx115_com_ioctl

vx115_com_stop

vx115_com_tty

vx115_com_poll

consdev functions

The consdev struct contains pointers to methods used for console
input and output. These functions are used for low-level operations like
kernel print statements. For example, the kernel printf function uses
the following call sequence:

The functions are very low-level routines that just directly write
to or read from the serial interface.

vx115_com_cngetc

spin until there's a character in the receive FIFO,
then read and return the character

vx115_com_cnputc

spin until there's room in the transmit FIFO, then
write the supplied character to the FIFO

vx115_com_cnpollc

Interrupt routines

The interrupt routines handle the actual transfer of data between
the driver's buffers and the serial interface FIFO. The hardware
routines transfer between the hardware FIFO and the buffers, while the
soft interrupt routines interpret the data and finish transmission as
needed.

Transmit

Transmission is done primarily through the hardware interrupt
routine, which is called whenever the transmit FIFO level falls below
a threshold. The soft interrupt routine is scheduled only when the
transmission is complete, and just synchronizes the tty buffer
pointers to account for the transmitted data and checks to see if new
data has become available for transmission, restarting transmission if
so.

vx115_tx_hard

while the transmit FIFO's not full

copy data from the tty output buffer to the FIFO and
update the softc count and buffer pointer

if the count is 0, transmission's complete

turn off the transmit interrupts

clear the sc_tx_busy flag, which will cause the soft
interrupt routine to run

vx115_com_txsoft

if the sc_tx_busy flag is clear, transmission is
complete

call ndflush to update the tty buffer pointers to
account for the data transmitted

call the tty t_linesw->l_start method to initiate
a new transmission if new data is available

this just directly calls the driver's
vx115_com_start method

Receive

vx115_rx_hard

while the receive FIFO's not empty and the driver receive
buffer's not full

copy data from the receive FIFO to the driver's
receive buffer and update the softc count and buffer
pointer

if the receive buffer's full, turn off receive
interrupts

vx115_rx_soft

while the receive buffer has characters

read next character from receive buffer, check for
framing, parity errors

call the tty method t_linesw->l_rint to deliver
the character to the tty layer

update receive buffer pointers and count

turn on receive interrupts if they're off and there's now
room in buffer

Early initialization

Because it's desirable to have serial output early in the boot
process for debugging purposes during execution of the
initarm( ) routine, there's an early initialization of the serial driver
in the init_arm( ) function.

Character and Block Device Major Number
Registration

There's one additional step involved in getting devices supported by
NetBSD, and that's to include their device major numbers in the file
arch/arm/conf/majors.arm32. This involves including a line in this file of
the form

device-major <name> char <num> [block <num>] [<rule>]

The name must be the prefix of the cdevsw or bdevsw struct name in the
source file - or rather, the corresponding character device switch
structure variable must be obtained by appending "_cdevsw" to the name
used in this file (and block devices should append "_bdevsw").

For the vx115_com serial device here, the cdevsw struct is
named

vx115_com_cdevsw

in the source file vx115_com.c,
and the corresponding line added to the majors.arm32 file is

device-major vx115_com char 110 vx115_com

The major number 110 was selected arbitrarily as an available (unused)
major number.

Supporting the ARM926EJ-S Variant

The processor used in this SoC is a variant of the generic ARM9
core, the ARM926EJ-S. A few tweaks were needed in the generic ARM
configuration code to provide support for this variant. Note that this is
nothing like providing full support for any new features in a variant
core, but just getting the core to be recognized as an ARM9 variant by the
system. Note also that the latest version of NetBSD fully supports this
processor, so these mods are no longer required.

arch/arm/arm32/cpu.c

a change was propagated from a more recent kernel version to
wtnames[14]:

"**unknown 14**" to "write-back-locking-C"

arch/arm/include/armreg.h

the identifier for this ARM9 variant was added to CPU arch
type defines:

CPU_ID_ARCH_V5TEJ 0x00060000

the ID for this variant was added to CPU ID defines:

#define CPU_ID_ARM926EJS 0x41069260

Observations and Comparison with Linux
Porting Efforts

Porting NetBSD to this SoC turned out to be a very straightforward
effort, with very few "potholes". I've done a few Linux ports before, and
the adaptation of NetBSD went at least as smoothly as any of the Linux
efforts. In fact, I found NetBSD to have several advantages that made it
substantially easier to port to a new platform than Linux, especially for
a "newbie" like myself with no previous NetBSD porting experience.

NetBSD builds its own NetBSD-specific toolchain as part of the
build process. With Linux, you need to acquire or build yourself a
toolchain that's appropriate for the processor architecture and C
library variant you're using. While there are a number of nice aids
for this process (e.g., Dan Kegel's crosstool), it can still be a
significant pain - for example, use the wrong binutils variant and
things break in mysterious ways. With NetBSD, just run the included
build script and the toolchain's there.

The NetBSD source includes the full system source - kernel and
applications. While this can reduce the flexibility somewhat (for
example, there's only one C library provided), it makes porting and
building much easier. With Linux, one of the main initial tasks is
pulling together the source for the components you want in your root
filesystem, and creating the build tree and scripts to create the
desired filesystem images. On the flip side, however, there are some
Linux distributions available (both free and commercial) that
provide full system source and a build environment - for example,
the Snapgear distribution, or the MontaVista offering - and one
advantage these have is that they generally provide a broad
selection of library and application choices, and tend to be more
easily configurable than the standard NetBSD distribution in
selecting the components to be included.

However, NetBSD has a few drawbacks (some significant),
particularly for use in embedded systems.

There is at the time of this writing no Flash Translation
Layer or dedicated Flash filesystem that provides journaling or
wear-leveling to support Flash memory (NOR or NAND). You need to
have the root filesystem on a disk or embedded as a RAM filesystem
in the kernel image (or mounted via NFS). Since the embedded space
is extremely Flash-centric, this is a significant hole. There are
apparently efforts to correct this, but in early 2007 it's not there
yet. (With the recent release of Apple's iPhone, which runs OSX,
whose kernel, Darwin, is BSD-based, and which uses Flash as its
storage medium, it looks like there is some solution that exists,
but whether this is open-source at this point I don't know.)

The NetBSD kernel doesn't provide the option to build itself
as a self-extracting compressed image (a zImage in Linux parlance).
The kernel image can be built as a compressed image, but it relies
on the bootloader to be able to uncompress it. However, NetBSD does
provide a utility called gzboot, which decompresses a gzipped kernel
to RAM and jumps to its entry point. This provides essentially the
same functionality as the Linux zImage. (Thanks to Allen Briggs for
the gzboot tip.)

Both Linux and NetBSD benefit tremendously from being open-source
systems. As noted in the introduction, it's because of the work of others,
and their openness in sharing it, that NetBSD and Linux and most other
open-source systems are quite straightforward to adapt to new platforms.
And there's nothing quite so satisfying as finally seeing on your console
(debug messages and warts and all):