When you say "anything", have you tried different file types? I'm using a custom .svg icon and it displays perfectly nicely regardless of panel size/mode (including deskbar mode) etc. On the other hand, I've just tested a .png icon, and that displays correctly too, so perhaps you have a bug somewhere. If no amount of fiddling with the panel settings fixes it, maybe try rebuilding xfce4-panel?