You can be a kernel hacker!

This blog post is adapted from a talk I gave at Strange Loop
2014 with the same title. Watch the video!

When I started Hacker School, I wanted to
learn how the Linux kernel works. I’d been using Linux for ten years,
but I still didn’t understand very well what my kernel did. While
there, I found out that:

I hadn’t been doing low level programming at all – I’d written a
little bit of C in university, and otherwise had been doing web
development and machine learning. But it turned out that my newfound
operating systems knowledge helped me solve regular programming tasks
more easily.

I also now feel like if I were to be put on Survivor: fix a bug in my
kernel’s USB driver, I’d stand a chance of not being immediately
kicked off the island.

This is all going to be about Linux, but a lot of the same concepts
apply to OS X. We’ll talk about

what even is a kernel?

why bother learning about this stuff?

A few strategies for understanding the Linux kernel better, on your own
terms:

strace all the things!

Read some kernel code!

Write a fun kernel module!

Write an operating system!

Try the Eudyptula challenge

Do an internship.

What even is a kernel?

In a few words:

A kernel is a bunch of code that knows how to interact with your hardware.

Linux is mostly written in C, with bit of assembly. Let’s say you go
to http://google.com in your browser. That requires typing, sending
data over a network, allocating some memory, and maybe writing some
cache files. Your kernel has code that

interprets your keypresses every time you press a key

speaks the TCP/IP protocol, for sending information over the network
to Google

communicates with your hard drive to write bytes to it

understands how your filesystem is implemented (what do the bytes on
the hard drive even mean?!)

gives CPU time to all the different processes that might be running

speaks to your graphics card to display the page

keeps track of all the memory that’s been allocated

and much, much more. All of that code is running all the time you’re
using your computer!

This is a lot to handle all at once! The only concept I want to you to
understand for the rest of this post is *system calls. System calls
are your kernel’s API – regular programs that you write can interact
with your computer’s hardware using system calls. A few example system
calls:

open opens files

sendto and recvfrom send and receive network data

write writes to disk

chmod changes the permissions of a file

brk and sbrk allocate memory

So when you call the open() function in Python, somewhere down the
stack that eventually uses the open system call.

That’s all you need to know about the kernel for now! It’s a bunch of
C code that’s running all the time on your computer, and you interact
with it using system calls.

Why learn about the Linux kernel, anyway?

There are some obvious reasons: it’s really fun! Not everyone knows
about it! Saying you wrote a kernel module for fun is cool!

But there’s a more serious reason: learning about the interface
between your operating system and your programs will make you a
better programmer. Let’s see how!

Reason 1: strace

Imagine that you’re writing a Python program, and it’s meant to be
reading some data from a file /user/bork/awesome.txt. But it’s not
working!

A pretty basic question is: is your program even opening the right
file? You could start using your regular debugging techniques to
investigate (print some things out! use a debugger!). But the amazing
thing is that on Linux, the only way to open a file is with the
open system call. You can get a list of all of these calls to open
(and therefore every file your program has opened) with a tool called
strace.

Let’s do a quick example! Let’s imagine I want to know what files
Chrome has opened!

It’s (deleted), but we can still look at it!
cat /proc/8604/exe > recovered_smile will recover our executable. Wow.

There’s also a ton of other really useful information about processes
in /proc. (like which files they have open – try ls -l/proc/<pid>/fd)

You can find out more with man proc.

Reason 3: ftrace

ftrace is totally different from strace. strace traces system
calls and ftrace traces kernel functions.

I honestly haven’t had occasion to do this yet but it is REALLY COOL
so I am telling you about it. Imagine that you’re having some problems
with TCP, and you’re seeing a lot of TCP retransmits. ftrace can give
you information about every time the TCP retransmit function in the
kernel is called!

Convinced yet?

Understanding your operating system better is super useful and will
make you a better programmer, even if you write Python. The most
useful tools for high-level programming I’ve found strace and
/proc. As far as I can tell ftrace and perf are mostly useful for
lower-level programming. There’s also tcpdump and lsof and
netstat and all kinds of things I won’t go into here.

Now you’re hopefully convinced that learning more about Linux is worth
your time. Let’s go over some strategies for understanding Linux
better!

Strategy 1: strace all the things!

I mentioned strace before briefly. strace is literally my favorite
program in the universe. A great way to get a better sense for what
your kernel is doing is – take a simple program that you understand
well (like ls), and run strace on it.

This will show you at what points the program is communicating with
your kernel. I took a 13 hour train ride from Montreal to New York
once and
straced killall and
it was REALLY FUN. Let’s try ls!

I ran strace -o out ls to save the output to a file. strace will
output a
WHOLE BUNCH OF CRAP.
It turns out that starting up a program is pretty complicated, and in
this case most of the system calls have to do with that. There’s a lot
of

and a bunch of other things I don’t really understand. My main
strategy when stracing for fun is to ignore all the crap at the
beginning, and just focus on what I understand. It turns out that ls
doesn’t need to do a lot!

That was really simple, and we already learned a new system call! That
mmap in the middle there? No idea what that does. But it’s totally
fine! STRACE IS THE BEST.

So! Running strace on random processes and looking up the
documentation for system calls you don’t recognize is an easy way to
learn a ton!

Warning: Don’t strace processes
that you actually need to run efficiently! strace is like putting a
huge stopsign in front of your process every time you use a system
call, which is all the time. Brendan Gregg has a
great post about strace which you should read.
Also you should probably read everything he writes.

Strategy 2: Read some kernel code!

Okay, let’s imagine that we’ve gotten interested in getdents (the
system call to list the contents of a directory), and we want to
understand better what it actually does. There’s this fantastic tool
called livegrep that lets you search through
kernel code. It’s by Nelson Elhage who
is pretty great.

So let’s use it to find the source for getdents, which lists all the
entries in a directory! I searched for it
using livegrep, and found
the source.

On line 211, it calls iterate_dir. So let’s look that up! It’s
here.
Honestly this code makes no sense to me (maybe res =
file->f_op->iterate(file, ctx) is what’s iterating over the directory?).

That’s it! printk writes to the system log, and if you run dmesg,
you’ll see what it printed!

Let’s look at another fun kernel module! I gave a talk about kernel
hacing at CUSEC in January, and I needed a
fun example. My friend Tavish suggested
“hey julia! What if you made a kernel module that rick rolls you every
time you open a file?” And my awesome partner
Kamal said “that sounds like fun!”
and inside a weekend he’d totally done it!

You can see the extremely well-commented source here:
rickroll.c.
Basically what it needs to do when loaded is

find the system call table (it turns out this is not trivial!)

Disable write protection so that we’re actually allowed to modify it
(!!)

SO FUN RIGHT. The source is super well documented and interesting and
you should
go read it.
And if you think “but Kamal must be a kernel hacking wizard! I could
never do that!“, it is not so! Kamal is pretty great, but he had never
written kernel code before that weekend. I understand that he googled
things like “how to hijack system call table linux”. You could do the
same!

Kernel modules are an especially nice way to start because writing toy
kernel modules plays nicely into writing real kernel modules like
hardware drivers. Or you could start out writing drivers right away!
Whatever floats your boat :) The reference for learning about writing
drivers is called Linux Device Drivers
or “LDD3”. The fabulous
Jessica McKellar is writing the
new version, LDD4.

Strategy 4: Write an operating system!

This sounds really unapproachable! And writing a full-featured
operating system from scratch is a ton of work. But the great
thing about operating systems is that yours don’t need to be full-featured!

I wrote a small operating system
that basically only has a keyboard driver. And doesn’t compile for
anyone except me. It was 3 weeks of work, and I learned SO MUCH.
There’s a super great wiki with
lots of information about making operating system.

I learned about linkers and bootloaders and interrupts and memory
management and how executing a program works and so many more things!
And I’ll never finish it,
and that’s okay.

Strategy 5: Do the Eudyptula challenge

If you don’t have an infinite number of ideas for hilarious kernel
module pranks to play on your friends (I certainly don’t!), the
Eudyptula Challenge is specifically
built to help you get started with kernel programming, with
progressively harder steps. The first one is to just write a
“hello world” kernel module, which is pretty straightforward!

They’re pretty strict about the way you send email (helping you
practice for the linux kernel mailing list, maybe!). I haven’t tried
it myself yet, but Alex Clemmer tells me that
it is hard but possible. Try it out!

Strategy 6: Do an internship

If you’re really serious about all this, there are a couple of
programs I know of:

Google Summer of Code, for students

The GNOME outreach program for women

The GNOME outreach program for women (OPW) is a great program that
provides mentorship and a 3-month paid internship for women who would
like to contribute to the Linux kernel.
More than 1000 patches
from OPW interns and alumni have been accepted into the kernel.

In the application you submit a simple patch to the kernel (!!), and
it’s very well documented. You don’t need to be an expert, though you
do need to know some C.

You can apply now! The application deadline for the current round
is October 31, 2014, and you can find more information on the
kernel OPW website.

Resources

To recap, here are a few super useful resources for learning that I’ve
mentioned:

Brendan Gregg has a ton of extremely
useful writing about performance analysis tools like perf and
ftrace on Linux.

You can be a kernel hacker

I’m not a kernel hacker, really. But now when I look at awesome actual
kernel hackers like Valerie Aurora or
Sarah Sharp, I no longer think that
they’re wizards. I now think those are people who worked really hard
on becoming better at kernel programming, and did it for a long time!
And if I spent a lot of time working on learning more, I could be a
kernel hacker too.