turn on vaapi USE flag, maybe. I just installed vlc and is not grayed out.

not related, vaapi and vdpau are two different none related implementations for gpu hw decoding.

Wallsandfences, check if opengl points to nvidia, not sure how this can work but on nvidia gpus, opengl must points to nvidia._________________Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
ProjectFootball

can you post the output of emerge --info vlc?
does it works with other players?_________________Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
ProjectFootball

set it to 1, also install mplayer and try playing the file using vdpau on mplayer_________________Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
ProjectFootball

AFAIK, "vaapi" is not limited to Intel-GPUs, even if it was developed for them. There's also an backend-module to use vdpau (Okay... An API that uses an API...), so that vaapi's more like a wrapper in this case.
So VLC uses vaapi to accelerate video-playback and vaapi just has to use vdpau. Works for me at home._________________Intel Core2Quad Q9450
MSI P35 Neo2-FR
4GB DDR2-800 RAM - Dual Channel
SB X-Fi Fatality Gamer
GeForce GTX 460
Gentoo Linux (kernel 2.6.x (was gerade stable im Portage ist ), amd64, Stage 1)
Windows 7 x64
Windows XP x64 SP2

vaapi has a wapper that converts it from vaapi calls to vdpau calls.
the poc of mplayer + ffmepg shows it, what is the gpu driver's version?_________________Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
ProjectFootball

I have diffitulties to understand all of the linked thread, and if it is related to my problem.
Further testing showed, that mplayer does not play all my videos. Also, is tough to set options, without interface. VLC seems to, but with high cpu load.

I disagree.. Because with all I know, totem should play hardware decoding, and mplayer should play all my files, no? VLC might not be able to use the graphic driver completly due to it's internal design or different desgin goals, ok.

VDPAU can also be used as a backend for the Video Acceleration API (VA API) which itself covers a subset of VDPAU's capabilities; so any software that supports VA API also partly supports VDPAU (e.g., VLC media player).