# Start the PulseAudio sound server in system mode.
# (enables the pulseaudio init script)
# System mode is not the recommended way to run PulseAudio as it has some
# limitations (such as no shared memory access) and could potentially allow
# users to disconnect or redirect each others audio streams. The
# recommend way to run PulseAudio is as a per-session daemon. For GNOME
# sessions you can install pulseaudio-esound-compat and GNOME will
# automatically start PulseAudio on login (if ESD is enabled in
# System->Preferences->Sound). For other sessions, you can simply start
# PulseAudio with "pulseaudio --daemonize".

If pulseaudio is executed as a per-user process, it uses shared memory to communicate; if it's started as a system daemon, it operates only by local sockets.
Obviously shared memory is faster than sockets, but unix local sockets are managed by the kernel in memory so they're not so slow as it seems.

To start pulseaudio as a daemon you have to:
1) Edit /etc/default/pulseaudio as written before
2) killall pulseaudio
3) sudo service pulseaudio restart

i've been smashing my head into this problem for the past few hours and i think i've got it figured out now. what a mess these stupid pulse config files are. try following these steps on a fresh pulseaudio install:

now put the following three lines in the bottom of your /etc/pulse/system.pa file:
load-module module-esound-protocol-tcp auth-ip-acl=127.0.0.1;192.168.0.0/16
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;192.168.0.0/16
load-module module-zeroconf-publish

you can remove the auth... junk from the first two lines there if you want, they just prevent any IP other than a local one from connecting to your server (although it might be cool for a few minutes if someone else started playing music on your speakers)

make sure you've got the pulseaudio-module-zeroconf package installed

now you can sudo etc/init.d/pulseaudio start and you should be up and running, and that should run on bootup

good luck

hopefully the time i've spent on this benefits someone other than just me

Even after this is all set up, how do I direct the sound devices? It's not at all intuitive. Whenever I start a server remotely, my local computer connects to it automatically and starts playing all sound through it. How do I route sounds to the local speakers by default and choose to move streams to the remote computer only when I want to?