NOTE: if your kernel version changed, you'll need to re-build the kernel again, with
$ make olconfig
$ make
In this case, you'll also need to re-install the drivers an the kernel with:
# make modules_install install

Removing the already loaded modules

This is probably the trickiest part. You can, of course, just reboot the machine, but this may take some time, so Mauro Chehab wrote a script that unloads all kernel modules from the memory. There's a rmmod.pl version that works with git and with all kernels that put the media drivers at the right location.

In order to remove all V4L/DVB drivers from the memory, all that it is needed is to run:

$ ./rmmod.pl unload

The other options of the script should be working as well, but I didn't test, since the only one I find really useful is the capability of removing the drivers from the memory.