There are two possible solutions: run PulseAudio with argument --high-priority=1 and make yourself member of the group pulse-rt, or increase the fragment sizes of the audio drivers. The former will allow PulseAudio to activate SCHED_FIFO high priority scheduling (root rights are dropped immediately after this). Keep in mind that this is a potential security hole!

I would like to avoid the high-priority solution if possible, so I tried number two.

Glad it worked. I got some help from two PA devs in the IRC channel, and I figured it was worth posting back. I provided them with as much info as I coud, but they unfortunately didn't entirely know if it was a driver or PA issue, so for now I believe the tsched workaround will remain in place.

In case it helps anyone else, here's it is again in more detailed, in /etc/pulse/default.pa

For reasons that weren't entirely clear to me, the pavucontrol application also forces down latency while it's running, possibly so it will have more up to date data-access for rendering the UI. So simply having it in the background is one workaround aswell.