Contents

kABI stability

A loaded kernel module is part of the Linux kernel and has access to all data structures
of the kernel. As such it needs to be compiled with exactly the right kernel configuration
the right compiler flags and -- most importantly -- the right kernel sources. In real life,
only the exported symbols are normally accessed -- but there are more than 6000 exported kernel
symbols in recent 2.6 kernels.

Due to the fact that kernel internal interfaces (kABI) are quite volatile and change frequently,
kernel updates can quite easily change some of the interfaces. This will break the kernel modules
that have been compiled for the older version. Fortunately this is detected by symbol versioning,
thus preventing the module from loading into the kernel and thus preventing data corruption.

Novell does actually spend quite some effort not to change the kABI in incompatible ways when
doing maintenance updates for the enterprise kernels; this does not change the fact that someone
providing kernel modules independently from the kernel provider needs to be prepared for kABI
change -- only much less often. Change happens when a service pack is released or possibly
when a critical bug or a severe security issue can not be fixed otherwise.

History

This close connection between the kernel and its modules made distributors (incl. SUSE/Novell) compile
them all together and provide them all in one binary kernel RPM. Every kernel module (mostly drivers)
that was needed by users needed to be included in the kernel, so it would be recompiled on every kernel
security update.

This approach of handling kernel modules has come to its limits:

Despite the willingness to accept modules in the tree, third party kernel modules existed, and no provisions were made to handle this correctly in case of kernel updates. In the worst case, this could lead to a system that was not bootable any more after a kernel update -- clearly not the best user experience.

Modules can not always be provided. There are legal restrictions, issues with code quality or portability or simply missing possibilities to test (be it due to missing hardware or due to missing manpower or knowledge).

Sometimes, kernel modules just are not ready in time a distribution is cut -- a way to provide modules/drivers later (or updated versions of existing modules) is highly useful then.

Enter Kernel Module Packages

The idea is to cleanly integrate kernel modules that are not part of the distributor-provided kernel RPM.
Anyone with the right knowledge can create a kernel module that matches a certain kernel version and works with it. The real difficult problem to solve is to maintain the module-provided functionality on a kernel security update.
This is achieved by reusing the module if possible and by requiring a recompiled modules if needed.

To address this, the following steps are taken on a kernel update:

detect third party kernel modules

for every module, check whether it still works with the new kernel (by looking at all the needed symbols and their versions)

if it still works, make sure it can and will be reused by the new kernel

if it is not compatible with the new kernel anymore

try to download a new kernel module RPM

if this is not available display an error message (and by default abort the update)

This is all that the user gets to see -- once kernel module packages are installed, updates will
automatically be downloaded and installed if needed.

But to make this happen, the distributor and the module provider need to do a bit of work:

To make sure new kernel modules get recompiled and provided on a server, the distributor can
offer to have people register their modules; the module creators then can be notified on kABI
changes that affect them. This is optional, but saves time as these notifications usually can be done
prior to a kernel release. Later, this should also be possible to do within the build
service provided by openSUSE.

Alternatively, the distributor can also offer to recompile the modules itself -- Novell
does do so for some extra modules it provides this way -- and
Novell Partner Engineering (formerly: Developer Services)
offers it as a service for partners -- see the Linux drivers page to get more information if this is interesting for your company.

In case of kABI changes, a mere recompilation of the module against the right kernel sources
does do the job -- it is Novell's policy not to require code changes (read do kAPI changes)
in security updates. This even applies to SLES service packs, though exceptions are not completely
impossible there. But as even this simple recompilation can create a bit of work, Novell does
watch changes closely nowadays and tries hard to avoid kABI changes for security/maintenance updates
for SLES and SUSE Linux. However, SLES service packs will typically change a good part of the
kABI -- and openSUSE factory kernels are expected to have kABI changes very often, as the upstream
development is tracked there.

Kernel Module Package format

There are two different mechanisms used to check compatibility of the module with the kernel,
depending on the version of SUSE Linux. For SLES9SP3/NLD9SP3/OESv1SP2 and SUSE Linux 10.0
(these versions subsequently referred to as Code9), there's a separate tool called kernel-update-tool
that does the work. It gets invoked on kernel updates and does all the checks and downloads.
For SLES10/SLED10/OESv2 and SUSE Linux 10.1 and later (subsequently referred to as Code10),
the solution has been integrated with the normal system installation, configuration and update tools.
Consequently, the dependencies on kernel symbols have been mapped to RPM dependencies that the dependency
resolver will then try to fulfill using the available installation/update sources and this way
automatically download needed modules or detect missing dependencies.

The kernel module packages are normal RPMs that contain the kernel modules (.ko files)
and a bit of metadata. For the third party kernel modules to be identified, updates to be
found, etc., the kernel module packages need to conform to packaging conventions.
Packaging conventions are documented separately for the older
Code9 (openSUSE 10.0 and SLES9) solution, for Code10 (openSUSE 10.1 and SLES10) solution, and for
newer one Code11 (openSUSE 11.1 and SLES11) solution.

The solution to provide downloadable updates is also slightly different.
Whereas the Code9 requires the RPM to arranged in a specific directory structure on a http (or ftp)
server, the Code10 and Code11 solution requires the format of a YaST installation source -- we recommend using
the yum format -- Code11 will also work with rpm-md repositories.

Like any other RPM, the kernel module packages should have a GnuPG signature. The updater
will check it and complain if the package is not signed, signed by an unknown party or is
corrupted. On initial installation, the install documentation or install script should thus
recommend to import the GnuPG signature key of the module package provider into the RPM keyring
to mark that RPM package provider as trusted. Key import should not happen in secret, but with
the explicit consent of the admin installing the package.

Initial Module Installation

The above deals with how third party kernel modules can survive kernel updates without creating
trouble for the user. But how can additional modules be found initially?

The module packages can be part of a product. Vendors may include the RPMs on driver CDs or whatever the normal software distribution channel is. For Code10, a tiny script that registers an update source and offers to import the GnuPG package signature key into the RPM keyring are recommended.

The module packages can be offered for download. RPMs can just be downloaded (Code9) or the download location can just be entered as installation and update source during installation time (or later) in the installer/updater (Code10/11).

The user can find the download sites by searching the vendor's site, reading documentation from the vendor or from others, following hints from openSUSE, search engines, etc.

To save search work, a better automated mechanism that saves the searching work for the end user using a software/driver portal where software providers could register additional installation/update sources which are offered by YaST optionally at installation time is in preparation. Offering packages as function of the detected hardware is also part of the design.

The packaging format specification also takes care that modules that have been created for
a different kernel version, but are compatible with the one that's installed on the
system will get used for the installed kernel.

Support and Certification

Support and Certification are only relevant for the Enterprise products (SLES, NLD/SLED, OES).
Support in this context refers to the ability of customers to open support calls and get guaranteed individual attention (including engineering efforts) to the problem they are reporting.

A kernel module has access to the kernel address space and can easily put the stability and reliability
of the operating system at risk. Modules unknown to Novell have the status "unsupported" and upon
loading them will generate a warning message to the syslog and "taint" the kernel.
If a user has such an unsupported kernel module loaded, Novell can not do very much to help with
kernel problems. Instead the problem should be reproduced without such a module loaded to be debuggable.

However, it is possible for a module provider to get the module in the "eXternally supported" status;
for this it's necessary that he works closely with Novell. The appropriate support channel
interfaces need to be in place to allow Novell support and the module provider to work jointly on
a customer issue. The module provider ust commit to maintain the module for the lifetime of SLES.
The kernel module package needs to be built according to the rules specified above -- best by
having it built by Novell's build service (Novell Partner Engineering). Modules built
this way can also be tested and certified and thus become part of certified systems.

Specfile mechanisms

In the specfile preamble, request that kernel-syms (pulls in kernel-devel) and modutils be present.

Requires: kernel-syms modutils
SourceN: %name-kmp-preamble

Right before %description, specify this to have kmp subpackage produced:

%kernel_module_package -p %name-kmp-preamble -x ec2 xen

The -p argument specifies additional rpm tags to be added to each kmp subpackage. The -x flag, if provided, requests the exclusion of these kernel flavors from the build. The reverse is omitting -x and just specifying flavors to which only build for.

A few tags are sourced from the tags of an otherwise empty "KMP" subpackage:

In the kmp preamble file, %1 denotes the flavor, and %2 the package version. In this file, there may be one or more of the following lines:

Enhances: kernel-%1 “Enhances” is a reverse “Suggests”. The mypkg-kmp-flavor subpackage may be suggested during the installation of kernel-flavor.

Supplements: packageand(kernel-%1:%name) “Supplements” is a reverse “Recommends”. The installation of the mypkg-kmp-flavor subpackage will be recommended if kernel-flavor too will be or is present in the system.