Note that the rtl-sdr driver will by default transfer data from the device in large chunks. In gqrx this can have the side effect of reduced time resolution in the waterfall or slow update rate of the pandapter. If you experience this you can try adjusting the buffer size in the device string by adding buflen=xyz, i.e. if you current device string is rtl=0 the new device string becomes rtl=0,buflen=xyz.

On the other hand, you always want to use the largest possible buffers because they often give the best performance (lowest CPU load). Buffer sizes that work for me at various sample rates: