QOS on VoIP without using MAC address

After some testing and a lot of searching, it seems it is not possible to use QOS in tomato to prioritize VoIP calls without using the MAC address of the device placing the call. Is this correct?

Simply prioritizing SIP only gives priority to the setting up and tearing down of the call, but not the actual call itself. This is very evident while looking at the QOS graphs. I want to give priority to calls placed from softphones (X-Lite in this case). If I could even prioritize based on the destination hostname, that would be an adequate workaround.