"Linux system calls - we use them every day. But do you know how a system call is performed from user-space to the kernel? Explore the Linux system call interface, learn how to add new system calls (and alternatives for doing so), and discover utilities related to the SCI."

I enjoyed this article much. Very interesting, allthough it might contain nothing the average user Joe Q. Sixpack is interested in. But I'm sure he won't read OSNews. :-)

"I own a few books about kernel development but I always enjoy these articles. Most of the time, they are more easy to follow than the books."

I really would like to see Linux kernel having a good documentation base; this article shows that it cannot be that hard. The last time I checked, it was hard to find out what certain kernel functions would to, "man kernfoo" did not give any results, so I had to read through the kernel files in order to find some comments. Just to mention a good example, the FreeBSD man pages contain very good information about the principal kernel functions.

If you liked this article, you'll love the instant classic narrative on the design and implementation of the Linux 2.6.x kernel: Linux Kernel Development, Second Edition, by Robert Love. This book is the perfect balance of the implementation details (with plenty of code listings) and a high-level discussion of the design considerations. It's easy to read cover-to-cover, never reading like a reference manual. Robert describes the kernel from the perspective of a long-time kernel hacker, discussing the ins and outs of the unique Linux kernel development community and how to properly submit patches. As long as you know the basics of C programming, you will have an excellent understanding of the core subsystems of the Linux kernel after reading this book.

"
1) The Kernel Documentation is always out of date. That is because the Linux kernel is a work in progress and probably will be until 2.6.30 sometime."

This might have been true during the old stable/development split model. With 2.6 development happens in -mm tree which is a soft fork and major features get added to every release which means the rate of change is always high and 2.6 will always be "work in progress" as you call it.

"2) I suspect once the 2.6 API is frozen and work begins on 2.8 or 3.0, documentation will firm up.
"
Linux kernel internal modules API never freezes. User space API and ABI is always stable. With the 2.6 development model dont expect a 2.8 or 3.0 anytime soon. It might never happen unless somone manages to come up with a change so far that cannot be incrementally pushed in and deserves a complete fork.

1) Completely true. There's some very helpful nuggets of wisdom in the Documentation directory, but many of these files are out-of-date, and large portions of the kernel simply aren't documented in plain English. However, the syscall interface and the key in-kernel APIs are well-documented.

The Linux philosophy is that the source code is the best documentation, and the high standards for coding style reflect this idea. I've hacked on a few UNIX kernels, free and proprietary, and the Linux kernel source is by far the cleanest and most deliberate kernel code that I've seen. The maintainers understand that simple code often outperforms sophisticated code, a lesson that my current employer has yet to learn...

Some outsiders complain that the code reviewers on LKML nitpick stylistic issues more than anything else. They do this for a reason. Code reviews can catch bugs, but they don't catch them all. A good code reviewer might find one out of every three bugs over a long period of time. Patch submissions will be broken from time to time, and often they will break other things.

The important things is that we are able to find and fix bugs when encountered. When the maintainers commit patches to the mainline, they aren't guaranteeing that the code is flawless, but they are guaranteeing that the code quality is high enough that the community can maintain it even if the original developer gets hit by a bus. That's why the Linux kernel maintainers are sticklers for style. They would rather have well-written code with some bugs than flawless code that nobody really understands.

2) Linux will probably never have a stable in-kernel API. Not now, not at some point in the 2.6.x series, and probably not in any later series. Linux is developed in the open, completely transparent to anyone who wishes to support it. They don't want to freeze the in-kernel APIs, limiting our ability to innovate and improve, just so that proprietary vendors don't have to monitor the LKML from time to time. If any vendor is unsure of how a planned API change will affect their driver, the kernel community will be glad to discuss the changes and recommend solutions. For example, the kernel community prepared extensive tutorials and documentation when the workqueue API was introduced in the 2.6.x kernel.

Having very stable APIs actually makes change more disruptive. Look at how many devices and applications don't function correctly on Vista. How could it be that Creative (for example) was caught off-guard by an OS release that was 2 years late? Were they expecting to grab a copy of Vista RTM, boot up, and have their audio drivers magically work with the new driver APIs? Many hardware vendors are just grossly negligent. There's no way to develop an OS that keeps their drivers working without staying stuck in the Dark Ages, so we don't make any attempt to do this in Linux. If you give us the code, we'll make sure it stays up-to-date. If you refuse, then the best we can offer is complete transparency and an open forum for communication.

Haven't found one that exists, so the only search I have of use right now is google.

I recommend cscope, ctags, or, if you're feeling ambitious, OpenGrok. Use google to find them ;-)