A Voyage to the Kernel, Day 11

We’re entering a new phase in our journey—kernel programming. In the first part, we’ll cover a broad introduction to the Linux platform for newbies, along with some history.

The Linux platform

The UNIX platform showed us the power of the multi-layer security architecture, and the beauty of a well-organised structure in terms of subsystems and layers. This is perhaps one aspect that inspired Linus Torvalds, who was then studying computer science at the University of Helsinki, to develop a free operating system. (It should be noted that there is still an ongoing project to remove the non-free portion of the kernel code.)
In 1990 he wrote:

The 90s home PCs were powerful enough to run a full-blown UNIX OS. So Linus thought it would be a good idea to come out with a freely available academic version of UNIX. He based his project on Minix. To quote him again:

Date: Aug 26 1991, 11:12 am
Subject: What would you like to see most in minix?
To: comp.os.minix

Hello everybody out there using minix -

I’m doing a (free) operating system (just a hobby, won’t be big and
professional like gnu) for 386(486) AT clones. This has been brewing
since april, and is starting to get ready. I’d like any feedback on
things people like/dislike in minix, as my OS resembles it somewhat
(same physical layout of the file-system (due to practical reasons)
among other things).

I’ve currently ported bash(1.08) and gcc(1.40), and things seem to work.
This implies that I’ll get something practical within a few months, and
I’d like to know what features most people would want. Any suggestions
are welcome, but I won’t promise I’ll implement them :-)
Linus (torva…@kruuna.helsinki.fi)

PS. Yes – it’s free of any minix code, and it has a multi-threaded fs.
It is NOT portable (uses 386 task switching etc), and it probably never
will support anything other than AT-harddisks, as that’s all I have :-(.

He wrote the first Linux kernel in 1991. Two years after Linus’ post, there were 12,000 Linux users. Linux is an essentially full UNIX clone. And now the Linux kernel is over six million lines of code! The main reason why Linux gained its popularity is because it was released under the GNU GPL licence (arguably one of the strongest ‘copyleft’ or quid pro quo licences). Later, Slackware, Red Hat, SuSE, Debian, et al, sprung up to provide packaged Linux distributions. This further spurred the growth of the GNU/Linux community.

Some of the earlier large-scale users include Amazon, the US Post Office and the German Army. It is interesting to note that Linux machine clusters were used while making movies like Titanic and Shrek. Today, we have Linux on gadgets like PDAs, mobiles, embedded applications and even on wristwatches. The introduction of 3D acceleration support, support for USB devices, single-click updates of systems and packages made Linux more popular. Desktop users can now log in graphically and start all applications without typing a single character on a terminal. At the same time, you still have the ability to access the core of the system.

Here are a few other utilities and programs (mostly from GNU) that added power to Linux:

Bash: The GNU shell

GCC: The GNU C compiler

coreutils: A set of basic UNIX-style utilities

findutils: For searching and finding files

GDB: The GNU debugger

fontutils: For converting fonts from one format to another or making new fonts

Emacs: A very powerful editor

Ghostscript and Ghostview: An interpreter and graphical front-end for the PostScript files

GNU Photo: A software to manipulate the inputs from digital cameras

Octave: To perform numerical computations

GNOME and KDE: GUI desktop environments

GNU SQL: A relational database system

Radius: A remote authentication and accounting server

An introduction to the tech side

The Linux kernel is monolithic with module support. Beginners may not fully appreciate the meaning of this statement. A monolithic kernel is essentially a single large complex “do-it-yourself” kernel program, which has several different logical entities, and these run in kernel mode. The Linux kernel has a modular design. When you boot the system, only a minimal resident kernel is loaded into memory. When you request a feature that is not in the kernel, a kernel module (or driver) is dynamically loaded into memory. The object code normally has a collection of functions that implements a file system, a device driver or any other feature at the kernel’s upper layer. This has the following advantages: minimal main memory usage, modularised structure and platform independence.

If you are an experienced programmer, then by looking at the dependencies you will know how the new file has been created. Novice users need not worry, as this column will address everything from scratch. For the time being, you can just explore the directories /lib/modules/2.x.x-version and /usr/src/linux-headers- 2.x.x-version.
As I mentioned before, you can load a module by issuing commands. lsmod will list the modules (with a small description about its current usage) in your terminal. Figure 1 shows the result of the execution.

Figure 1: A typical lsmod output

You can also view the list of modules to load at boot time by checking the /etc/modules file. It will have entries similar to the one listed below:

As this is an introductory column, the kernel build process (which has five parts, viz. Makefile, .config, arch/$(ARCH)/Makefile, scripts/Makefile, Kbuild Makefiles) will be explained in detail in subsequent articles.

Compiling the kernel

Now we will look at how to compile a kernel:

Download the latest Linux source. If you have installed Linux with the kernel source option, the source will already be there. You may copy the kernel source to a safe location so that you can restore it later. It is a good idea to backup your /boot directory as well or at least the kernel image. On my machine, the source was located in /usr/src/linux-2.6.27-7.

Create a soft link to your Linux folder, so that you have /usr/src/linux:

ln -s /usr/src/linux-2.6.27-7 /usr/src/linux

Make the necessary changes to the kernel

make clean

make mrproper

make clean

Configure the modules. You have to specify the modules to be loaded and those to be compiled in the kernel. It would be ideal to start with your existing configuration file.

make dep

Modify your Makefile EXTRAVERSION tag to give a unique name to your kernel

(Optional step) Delete the entry for the old kernel version in the config file

Reboot. When the machine restarts, you can select the new kernel!

This is the standard procedure. There are simpler methods as well. (For example, in the case of Debian-based distros, you have a very easy method. Just Google it and find the resources.)

Kernel structure

Within the kernel layer, Linux is composed of five (or six, based on the classification style) major subsystems: the process scheduler (sched), the memory manager (mm), the virtual file system (vfs), the network interface (net), and the inter-process communication (ipc).

Organisation of code

We shall now discuss the way in which the source layout is organised. If you go to the root of the source tree and list the contents, you will see the following directories:

arch – Contains architecture-specific files

block – Has the implementation of I/O scheduling algorithms

crypto – Has the implementation for cipher operations and contains the cryptographic API for implementing encryption algorithms

Documentation – Contains descriptions about various kernel subsystems

drivers – Device drivers for various device classes and peripheral controllers can be found here

The author is a hacker and a free software activist who does programming in the open source domain. He is the developer of V-language -- a programming language that employs AI and ANN. His research work/publications are available on his website.