NOTE: if your kernel version changed, you'll need to re-build the kernel again, with
$ make olconfig
$ make

Removing the already loaded modules

This is probably the trickiest part. You can, of course, just reboot the machine, but this may take some time, so Mauro Chehab wrote a script that unloads all kernel modules from the memory. There's a rmmod.pl version that works with git and with all kernels that put the media drivers at the right location.