Do you use the open source drivers and have VDPAU enabled for mesa? If so, disable it.

Only if you have a kernel which is patched for UVD and >=mesa-9.2_pre20130404 then VDPAU becomes useful. Else it will attempt to decode certain video formats using shaders with often suboptimal results.

It could be that mplayer chooses a suboptimal video output (VO). Check the messages that are printed by mplayer to the shell. With the radeon driver and in absence of UVD acceleration, -vo xv is preferable.