The (full duplex) directx drivers will give you huge latency, e.g. with the Soundblaster Live it's 750 (!) ms.

The default buffer setting of my Audiophile gives 8 ms, which is totally ok. Anything with a latency lower than 20 ms is something you can work with, single digit latency is desireable though.
I can even get lower than 8 ms, but it's a good compromise between speed and stability (I only have a PIII 800).