Posted
by
kdawson
on Tuesday July 03, 2007 @09:02PM
from the those-who-teach dept.

hdxia writes "Can anybody recommend a good Linux kernel training course? I have had some Linux kernel hacking experience, but would like to further harden and improve my understanding of the kernel. I expect the course would be advanced. You may say that the best method would be to dig into the kernel myself, but I really want to have a chance to discuss and learn all aspects of the kernel with an experienced instructor."

Kernel hacking is taught at major universities, but under the title "Operating Systems". These classes teach scheduling, device drivers, file systems, memory management, networking, etc. The goal of a university is to give you the background knowledge necessary to understand specific implementations. The specifics of the linux kernel change rapidly, but the concepts are 40 years old. No instructional course has a chance of keeping up, the kernel documentation cant even keep up. It takes a problem solver, and an inquizitive mind to be able to apply the background knowledge to specific instances.

I like to say theory>implementation. It's why I have more respect for Linux S/A's more than Windows S/A's - Linux exposes you and sometimes forces you to understand "theory" but the Windows Wizards only give you implementation.

Kernel hacking is taught at major universities, but under the title "Operating Systems".

We did no kernel programming in either the undergrad or graduate OS class at my undergraduate institution (I took both). It was all fairly low-level C systems programming. The undergrad OS class at my graduate institution (a top ten CS school) is the same way -- in fact, the OS class there is sometimes taught in Java. We did no kernel programming in any class I've taken so far, except for the graduate OS class at my graduate institution, and that was only because that was the project I happened to pick.

You learn OS theory, sure, and you almost need an understanding of a lot of that to know all of what's going on around the Linux kernel, but you're not going to learn about the Linux kernel in an OS class.

We did no kernel programming in either the undergrad or graduate OS class at my undergraduate institution (I took both). It was all fairly low-level C systems programming. The undergrad OS class at my graduate institution (a top ten CS school) is the same way -- in fact, the OS class there is sometimes taught in Java.

I'm currently the 3rd year Undergraduate Operating Systems instructor at a big University, and while I'd love to have my students do their work with a real kernel (and preferably an Open Source kernel, like Linux, BSD, or Xnu), it just isn't feasible. First off, the student's C is somewhat weak -- by the time they get to me, they've spent most of their time working in Java. Giving them 2 - 3 week long assignments hacking the Linux kernel would absolutely brutalize them. In my case, I have have a lab issue -- the department hasn't assigned a lab to the course, so I don't have a common system they can do their work on. And even if I did, the IT department probably wouldn't be too fond of them having the ability to recompile and load their own kernels (although this could be mitigated by having them run entirely within a VM -- if we had an assigned lab for the course). And finally, the burden on the grad student marker to be able to mark such assignments would be rough.

The only way I can think to make such a course work (at least where I'm teaching) would be to ask the students to study and explain how various OS subsystems work. I'm all for doing such a thing, but my department wants the students to do programming assignments (note that I'm just a lowly Instructor -- I'm not a tenured Professor).

In the end, however, I don't think that it's realistic for me to expect my students to be able to write an OS once they get out, as few (if any) ever will. My stated goals for them are to have them be able to understand how OS's work, so that they can a) write code that interfaces correctly with the system (API/system calls, IPC, memory management, etc.), and b) be able to compare and contrast different aspects of different OS's, and recommend the best OS for a given task.

Now ideally, my course would then segue to a more advanced kernel-hacking course for those who are sufficiently motivated to take it. However, I doubt many Universities have a suitable practical kernel hacking Instructor/Professor on staff. I'd love to be able to teach such a course, but my practical Linux kernel experience doesn't really extend beyond make clean;make menuconfig;make dep;make;make modules;make install.

Such a course would be cool -- I just imagine many Universities lack the expertise in house to offer such a course.

The third year OS course I took had us writing schedulers etc for a simulated Operating System. Maybe something of that sort would be good. It was very handy as the fourth year real-time systems class I took had us writing a scheduler with hard timing constraints in assembley for a robot.

I had the complete opposite experience at my university. Our instructor was fantastic and INCREDIBLY knowledgeable - he actually wrote part of the original BSD kernel back in the day. We were lucky because they did set up computers for us to play with - dual boot windows and freeBSD. Students were given the root password for the freeBSD, with the full knowledge that the installs WOULD be messed up and and have to be repaired pretty frequently. We didn't have individual accounts on freeBSD (used linux in

I am a bit surprised with what I read, as I happen to know many academic institutions (UK based) that do teach the inner parts of the Linux kernel to final year undergrads. Some courses are a little bit out of date (teaching 2.4 kernel stuff) but one could get the basic idea and then find references (see latter paragraphs) to jump from 2.4 to 2.6 . In fact, I graduated from one of them 7 years ago and I do remember the long nights behind a source editor looking at kernel structures and making the kernel pan

I fear for the future of humanity if all Universities are doing is churning out mediocre Java developers and then cutting out essential parts of the course because it doesn't fit the Javarific world they've chosen for themselves. Tannenbaum would be spinning on his bicycle.

I don't disagree at all. By the time they get to me, they will have had a little bit of C experience, but it's just a perhaps month-long module in a course on general Unix development (which also includes Unix commands, Perl, and CVS)

My experience has been the exact opposite. I took an operating systems course at a four year university, and we didn't write a single line code that wasn't in user land. That wasn't the intro to operating systems course either, that was the 400 level five credit course. I even went to

The replies to this comment seem to be along the lines of "they don't teach that at my university, so you're wrong." As a clarification to the parent, it should be noted that the CS 503 [purdue.edu] course taught at Purdue (cide1's and my university) has a lab where you have to implement a scheduler, a full virtual memory subsystem with demand paging, and a file system in the Xinu operating system on real hardware. If you break something, the only feedback you get is three beeps and a new boot prompt about 30 seconds

Prof. Raphael Finkel (about [wikipedia.org])has been offering a "Linux Internals" course for a few years now. He started it with the 2.4 kernel with Understanding the Linux Kernel (2nd Edition) by Bovet & Cesati as the text book; he's even updated the course to the 2.6 kernel and uses both Robert Love's Linux Kernel Development (2nd edition) and Bovet & Cesati(3rd Edition) as references. Check out the syllabus here:

Actually, depending on the school, there may be a course. I had a systems programming course where the prof chose the linux kernel as the subject. It could've been a little more advanced but there were a number of students that didn't care about linux or programming for that matter (it was a requirement for the degree).

OS design is covered at universities and the Linux kernel is often specifically examined. But they also show how other systems do things (and how they did it in the past). From what I gathered from my OS class, modern kernels are not really THAT different. I mean, they all have to do the same basic things. Once you learn the basics you should be able to just dive into the LInux kernel code and see how it works in detail.

Hogwash.
Gentoo is a source-based Linux distribution. A package manager.
As far as learning the system components, http://www.linuxfromscratch.org/ [linuxfromscratch.org] is a far better tutorial.
But neither is intended to be a kernel introduction.

Uh, what? You've never heard someone say that somebody has "a solid understanding" of a subject? It's a pretty common expression to refer to knowledge in that way, as if it were a physical thing - ie. to make your knowledge or understanding more tangible.

To easiest way to harden one's understanding is to take a normal soft understanding and add the cement of surety. Fill any holes in your understanding with a quick setting rigid factual putty until your mind is completely closed. Softening one's outlook and poking holes in facades has the opposite effect.

Another way to gain a hard understanding is to construct one's mind from steel. In this way one may obtain a mind like a steel trap, although over time and exposed to natural elements, such a trap will ine

Just start implementing in the stable kernel and as each of your inputs is rejected, you'll learn. I hope you've had - at least - an operating systems design class (or equiv experience), and you don't try to implement something in kernel space that should be in user space.BTW, I'm not qualified beyond hacking the IP stack a few years ago with a search/replace, use your imagination for what text was removed/replaced. all this, Just for Fun http://www.amazon.com/Just-Fun-Story-Accidental-Re volutionary/dp/0 [amazon.com]

That just isn't true! Experience should include a lot of independent research and planning. It isn't like you're just blindly trying things to see if they work or not. The only real difference between school learning and experience based learning is that there are real things at stake when learning by experience. And that can be a great motivator.

"The only real difference between school learning and experience based learning is that there are real things at stake when learning by experience. And that can be a great motivator."

It motivates you to go for the quick and easy solution instead of the right solution. And as the gp said, in the end that will cause you to go with the "test first, lesson afterwards (if at all)" approach.

Even in our young industry there is a wealth of knowledge out there on how to best do things. To throw it out in the

It motivates you to go for the quick and easy solution instead of the right solution. And as the gp said, in the end that will cause you to go with the "test first, lesson afterwards (if at all)"

It depends on the environment, how much pressure you're under, and how much you know going into the situation. What you say is true if you throw someone who doesn't know much into a high pressure situation with high stakes. They're going to find the easiest and quickest solution possible. No question about that. But if a person is consistently pushed only a little bit beyond their capabilities with reasonable demands and stakes, I'm confident that it can be a constructive learning experience. This is even employed in school. You're given reasonable tasks/projects to gain experience. But eventually you outgrow the the kind of experience that a school can offer. Eventually you need to go into the real world. You might start as an intern, or a junior programmer, for example, until you learn from enough experience to move on....

Even in our young industry there is a wealth of knowledge out there on how to best do things. To throw it out in the guise of "I'll experience it all myself" is just plain wasteful.

A solid education is important as a foundation. I'd never dispute that.

Yes, after one has gotten their foundation from formal education, or even in conjunction with formal education, real world experience is a benefit. No one is saying you should stay in school and never leave for the real world. However, that is different from saying things like "experience is the best teacher". This is important because (especially in computer programming) there are many out there who have no formal background in what they are doing and try to learn as they go, much like the original post

Truly "advanced" courses are hard to come by, because of limited appeal and other factors. The further you go into something the more inevitable specialization becomes, so it's not just a matter of offering one course for the tenth-of-a-percent of the market who's even heard of the Linux Kernel, but in all likelihood offering half a dozen ranging from security to device drivers to assembler optimization and so on. (and I'm just guessing here, knowing next to nothing about the kernel myself)

My suggestion would be to find someone who's pretty savvy in the area you're aimed at, and hire him or her (OK, let's face it, "him"...) for some lessons. Keep in mind that a good programmer is not the same as a good teacher, but if you find someone who can explain things the way you need to hear them then you won't need that many lessons to make a lot of progress - the cost could very well end up in the same league as a commercially-vended course.

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

When it comes to this level of specialization you take what you can get, and be happy about it. There is a reason many Universities have brilliant professors doing incredible research who also happen to be poor teachers. (At least in the sciences.)

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

Personal tutoring is a pretty expensive way to get an education, especially if it's in a fast-moving field whose experts are in demand for other work.

Off the top of my head:

Volunteer, and find a problem to work on.

Find someone to hire you to do kernel work, or some other way to work with people doing what you want to do.

Have you considered grad school? There's places where you could get a degree taking operating systems classes and hacking the kernel for your dissertation. And when you're done you'll probably have an easier time with the previous item.

Local user groups and universities might be good places to meet up with people who share your interests. Maybe start a study group to learn some kernel subsystem together?

Conferences: OLS [linuxsymposium.org] and linux.conf.au [linux.org.au] are fun. The OLS papers at least are on line if you can't go.

And if people have told you that "the best method would be to dig into the kernel myself",... actually, in the end, it's probably the *only* way. There's a certain point in your study of any field where you just run out of "courses". That's good. It means you're ready to do real work, because you're at the point where people are still busy doing the work and figuring stuff out, and haven't yet figured out how to break it down into manageable chunks and explain it in a logical order, which is a great deal of work in and of itself.

I don't know how fair it is to single people out--they probably just want to get on with their work without being made examples of. But Val Henson (chunkfs), Mingming Cao (ext3/ext4), and Suparna Bhattacharya (aio, various fs hacking) are three that come to mind immediately.

And the fact that you may not know their names if you aren't a kernel developer doesn't mean much. I don't think most pe

Many of the big names "outsource" their very technical training to third parties unless the topic is something related to an internal project (I won't name those companies here, but suffice to say that their names are abbreviations and usually three characters or less in length:)).

These training classes will cost the company $1k-$2k per student, depending on the exact nature of the course. This covers a 5-day class, 7-8 hours per day, with labs as practical for each topic. Obviously, in 5 days you're not going to get a lot of depth in any single topic, but a good instructor will be able to answer off-the-cuff questions during breaks between topics. I know that in a 10-15 minute break, I often get about 3 minutes to make a run to a restroom and spend the rest of the time going over details with one or more students. The most common areas of questions include processor scheduling (big changes in this area right now), virtual memory implementation details (especially the slab allocator and zoned memory concepts), and the block layer API. We don't get heavy into implementation details, but the student is expected to have at least some C background so that they can accomplish the lab exercises.

Some customers require detailed knowledge about specific subsystems and I will add a "chalk talk" ("dry erase talk"?) as time permits to cover those areas as much as possible. For example, a company that makes video poker machines running Linux might want details on hacking the interrupt handlers, while a company that builds disk storage units might want to talk about how best to support a custom RAID controller. Those types of things come up primarily in the Linux Device Driver course; the Internals course typically comes first in the curriculum and can be applied by application and system programmers to the code they write.

Most individuals are not going to be able to afford the associated costs, however. There are some training companies that offer "public" courses: I do those classes as well as on-site classes, but public classes for Internals don't often happen because there's not enough interest to fill the classroom with warm bodies. Send me a message if you're looking for such classes and I'll give you a list of vendors. If you opt to go the less expensive route, I suggest you get Robert Love's book on the Linux Kernel; overall it's a great book, but it does lack depth in one or two areas. After you understand what he covers there, then download the free book on the Linux VMM from

Yep, we're just like artisans. Toiling away without any standards. No codification of best practices. We even have our little cults: extreme programming, object and aspect oriented programming, spiral and waterfall methodologies, not to mention the plethora of programming languages. When programmers get old we don't put them into teaching positions to pass on their arcane knowledge, we make them managers and prohibit them from even looking at the code.

It's not surprising the software is so crumby, it's only surprising that it works at all.

The sad thing about it is that so many programmers think that what they're doing is wonderful and great and worthy of respect. We're not. This field is, at the moment, total crap, even at the highest and most "professional" end.

Yes, because we all know everyone that worked and innovated in newly created fields of knowledge were just in their dark ages too, and all they did was essentially crap because it was so new and they didn't have any previously-set standards to follow (the inconveniences of developing

Hello all, please give me, a foreign consultant with no work experience but an H1B visa, all your knowledge and a step by step instruction on how to do your job, so that I can displace you in yours, since your boss only looks at buzzwords on a resume anyways.I await your full attention to this matter,Samir Nagheenanajar

When I did tech support I worked with a lot of East Indians who would probably find that snippet quite amusing. I never did find humour offensive, but I realize that humour (along with any other art form) will always insight the Politically Correct.

It would be nice if I could feel free to express myself and give a bit of humour to the world without worrying about offending somebody. Unfortunately there will always be the intolerant and Politically Correct among us who will project there own anxieties on other people.

Say anything good, bad or neutral that involves a "race", and that statement can be said to be "racial", but "racist" implies intolerance and dis-respect, and this is something I just don't see. Perhaps this is because I view ethnic and phenomic differences as trivial, and yet view humour and the creative use of words and ideas as a very important part of my personality. The sad thing is, is that I don't use humour a lot because I know there will always be some people who just don't "get it". So I just give up:(

When I did tech support I worked with a lot of East Indians who would probably find that snippet quite amusing.

Just because people laugh at offensive jokes doesn't make them offensive. Hell if you were working in India and people made racist comments about your ethnic group, you'd probably decide that it was easier to laugh them off than risk losing your job. Ok, it's probably hard to explain to you, since if you were working in India you'd probably not be too unhappy about having to go back home where wage

To put things in perspective, from my experience most tech support type jobs are filled by Indians (I live in Canada), so I was always the visible minority where I worked. I mentioned that East Indians would likely find it humurous because my Indian co-workers were the types of people to circulate YouTube video's etc that make fun of the stereo-type of the typical call-centre worker being Indian. They also had a preference for "ethnic" humour, like Borat and others.The main premise of the joke in question i

We've all seen "please do my homework/job for me. fast response appreciated coz I has a dedline!" type posts to mailing lists. What's racist about the post is suggesting that this is somehow typical of foreigners.

In my experience Indians are disproportionatley represented in that group - and among the interview crib seekers too, I may add.

But I doubt anybody was suggesting they all do it, or that only they do it. Stereotypes aren't always true in every individual case, but ask yourself where they come fro

I'd be damned if I could find the correct spelling. Thanks. I realized my mistake after I posted, but still couldn't find the phonetic equivalent (spelling). But yes "insight" seems somewhat appropriate too:)

Well, good luck. Have you ever heard the saying, "Those who can, do; those who can't, teach" ? I don't agree with it , but there is a kernel of truth. You've probably had a professor who was a genius, and and expert in their field, but couldn't teach worth a damn. You've probably also learned from someone who was a good teacher, but didn't know their stuff, or didn't have the resources to teach it properly.

Finding someone who is an expert in the linux kernel, *and* who can teach, and has the time and willingness to teach you one-on-one, will be a rare find indeed. ( Are you willing to pay them what they're worth for their background and ability? )

Teaching is the best way to learn.
I was a computer science tutor up at Berkeley and I learned FAR MORE from tutoring than I did from the classes.
Find someone who is interested in the linux kernel and teach them. If they're smart they'll ask you questions you've never thought about asking, which in turn will end up solidifying your knowledge.

Actually, I think in this instance, even the GNU folks would call it just plain Linux. See, Linux is the kernel. GNU is the core software. GNU/Linux is what they want people to call the whole integrated operating system.

Many people will not spot the joke so I will point out that linux is not a gnu project.

There was a move a few years ago to get distributions of linux, gnu tools, X windows and other bits renamed to LiGnuX to draw attention the the gnu project. This wasn't taken seriously so later there was a move to put a gnu prefix on before linux in the name of distributions - only Debian took it up as far as I know. Newbies that missed the point also started putting gnu in front of any reference to a distribution conta

Out here on the left coast the extension programs at the various University Of California campuses have some Linux Kernel classes. These tend to be developed and taught by engineers in the industry with a real working knowledge of the subject.UC Santa Cruz Extension, http://www.ucsc-extension.edu/ [ucsc-extension.edu] has an "Linux Kernel Architecture and Programming" which looks like an intro course. You can take it online or as two Saturdays. There is also a Linux device drivers class which a once a week class and an Advanced Device Drivers class which is 4 Saturdays.I checked the other campuses but they all seem to be summer schedule with a limited set of classes.Red Hat also has a one week Kernel internals class which is a "hands on" which to me means a trade off of less information for some finger programing of the brain.All of these courses seem to have an introductory flavor to them. But I suspect that you will learn a lot about all of the various areas of the kernel and how the different parts hang together. My experience as kernel hacker is that I have learned a lot about the parts I am interested in, but that there are many big areas of the kernel that I only have a superficial understanding of.

...this made the front page? The course is called "Operating Systems," and it's taught at major universities, as someone previously mentioned. In many of these courses, the semester-long project is to develop your own distribution of Linux (as a class or group).

If what you're looking for is the typical corporate 1 week training course, then IMHO you can't do much better. But! Let's be honest here, what can you get out of a one week course? If it's a serious course, like these are, you're going to get so much information that you'll have trouble staying afloat. You're not going to master these skills in a week. Depending on your programming skills and operating systems background you're looking at months to years before you're comfortable kernel hacking. What these courses can give you is a lot of solid coding examples to build your skills. But don't take my word for it, the code samples are available at ftp://axian.com/pub/RHD_SOLUTIONS [axian.com]. Oh, and of course the Red Hat info: http://www.redhat.com/training/developer/courses/ [redhat.com]

If you're not in the typical corporate time crunch mode, I'd definitely recommend college courses. Get some general background in OS design while you're at it if you don't already have it. Oh, and LUGs, lots of lugs have groups doing kernel hacking, not a bad place to start there either, plus LUGs don't cost anything!

I doubt your target for course is correct. Having been trough after-studies Operating Systems course like 20 years ago, centering on RSX-11, I still feel solid in concepts of today's OSes - Linux, too.

For good professional background, I would recommend you to get also good books. On OS principles themselves, and on Linux Kernel. I have been printing and binding myself freely distributable online David Rusling book "The Linux Kernel" (that over 10 years ago). It mentions even Alpha processor view of things - very good for broader understanding. Love this book, very special. Also in my library sits book of Linux kernel anatomy: too lazy to go upstairs for exact title, but it must have been published by SAMS, and is book analyzing specifically code, that makes Linux Kernel, there are sources of very first versions of kernel included too, which should help to understand evolution of kernel as final touch. And plenty of code and discussion of it. Author might be not available for classes, but book is serious alternative for such.

It happened: went upstairs to check what was that other book, and it is translation of this one:"Linux Core Kernel Commentary: Guide to Insider's Knowledge on the Core Kernel of the Linux Code"by Scott Maxwell. Beware, that most of this book is made of kernel code, and one's expectation might be disappointed. However, it is very illustrative and should be handy for learning. There was once ago such "Linux Bible", published by Yggdrasil, consisting of many open source documentation of that time only - you co

ive never really heard of a hacking class or a kernel hacking instruction. ive gone through the usual computer security system classes in college to get that certificate, but what you go over is securing your system and how holes are taken advantage of. you don't learn how to find new holes.But since we're speaking of kernel hacking, that necessarily doesnt have to do with security, especially with what you're speaking of. the closest i can think of are book references to device driver coding and such, but

I recently finish a course that was titled "Linux Kernel & Device Driver Programming". This is by far the best class that I have taken in my college career. The book for the course was Linux Device Drivers, 3rd Edition which is available for free now. The course was more of a guide. It did not teach how how to program the linux kernel but instead how to research and a general overview and methodology of the kernel. The teachers showed us some of the the code and designs and why these designs were c

The code is pretty well commented, and it's usually worth having a good look at the sections you're interested in. It's well organised and clean [in the interesting parts]. It has to be, it's modified by a lot of people. Seriously, you won't be sorry. Above this, Google is your best friend. IBM's developer works has good holistic info, and Linux HQ [linuxhq.com] has lots of links to great information. Kernel hacking on your own time is pretty easy, I don't think it needs to be taught. If you want to work on THE Linux ker

A quick google of "Linux kernel training" brings up excellent resources on the first page. Choose from in-depth courses given by Red Hat, IBM, Sun, Linux Certified, and others. Combine that with self-taught hacking on the code and I think that would suffice to get you started.

There is quite a lot of training available about the Linux kernel. There is of course more to learn than can be possibly be taught in any courses, or can even be known by any one person.

A common issue in training is folks requesting training that covers more material than can be covered in the alloted time. However, when negotiating for training with a training provider, it is valuable to discuss the topic outline in advance and ask about changes. Since the topic of Linux kernel training is so broad, you

Failure comes wrapped in many packages. I'd hate to think my desire to have someone who could handle setbacks and failures would cause me to overlook a person who had failed in other careers, been kicked to the curb in other disciplines, and finally found their place in the sun, so to speak.

Every sorcerer was once an apprentice. Every wizard was once a n00b. Even Linus, Bill, and Steve.