Note: I no longer have the laptop in question, so unfortunately I can't test the answers anymore. Thanks for your effort anyway. This was nearly 3 years ago, and now I feel like it's unfair to accept my own answer to my question which doesn't even provide an answer, so I deleted the answer.

Before you set the VLC hotkeys, you need to set VLC as your default media player, in control panel, default programs, set your default programs, and choose VLC as default.

The global hotkeys should then work with VLC, you still have to set them in VLC - tools, preferences, hotkeys, double click on the global key you want to set, VLC will prompt you to press the key, and restart VLC afterwards so they become active.

None of the solutions above should work, as the problem is in the ATK Media utility provided by Asus. Instead of emiting keyboard events (like normal media keys), it maps them to specific apps (you can actually change those apps in the registry, see here http://mcbx.netne.net/hacks/asusmmed/index.htm). The order and name of apps that it looks for change depending on version and system, but luckily there's an easier fix to enable the use of generic media key events.

I just came across the solution for this after some time reading a bunch of sites.

Put DMedia.exe you'll find in the RAR into the ATK Media dir. Double click it to run it but it gets run at boot anyway.

Check if the media keys work as they should. Grab a beer or watch a movie if it works!

I just tested it in my Asus U30Jc under Win8 Pro 64-bits and works great! This tool actually maps generic keyboard events to the media keys in the integrated keyboard, so they work with any app enabled for this (in some cases like Winamp you might need to enable global keys).

After enabling hotkeys in VLC it should work with no problems, and with any other media program too (Spotify, WMP, Winamp, Xbox Music, etc).

Sorry for being unclear. In my first link I was referring to searching HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\AppKey to maybe find out if the associations for the multimedia keys are defined there.
–
harrymcFeb 3 '11 at 11:00

I just had this problem, and it turned out that iTunes advanced setting "Enable full keyboard navigation" was checked. I unchecked it, and the keys worked in VLC media player again, without having to map them.