It may be a config problem or a content problem. You can run alsamixer from a consle on the machine in question and check the supported options and settings. However the content may not cooperate with your desires- 5 ch in and 2 ch out unless to set it that way.

I have my SPDIF connected to my amp no problem, and the AVWizard plays Dolby Digital through it - I can see on the amp and heard that it is surround. But the DTS test doesn't work for some reason - I heard nothing at all! The amp indicates that it is getting no input. The exact same cable into the exact same amp from another source plays both perfectly.....

yeap, there is a compatibility issue, but that needs more investigation.

the DTS test works for Coax digital but for optical just wont play. the file is there but the receiver doesn't play it. the dts file is not making loud noise (i can't hear it playing) but at least the receiver locks on dts after dolby test.