The bigger the audio buffer size, the lower the chance you will hear unexpected clicks in the audio when using a lot of processing power. But at the same a bigger audio buffer size also causes a higher latency.

Using a smaller audio buffer size gives you a lower latency, but then make sure your CPU can handle it, otherwise clicks may appear in your audio because the CPU can't process it all quick enough.