I have a Q&A website with a sequencial registration/create session and the caching was delaying some seconds the creation of the user blocking the create session...and sometimes when having some simultaneous users showed the "misconfiguration server" splash screen that I couldn't find till now an associated pattern...
–
foxtrotDec 13 '10 at 18:50

Simple answer is - always. Cached site is simply faster than non-cached, that is better on all fronts - performance, resource consumption, user experience.

More complex answer - you should always use properly configured caching that fits your needs, size of site, traffic and available hosting resources.

For examples:

blanket static cache can do more harm than good in some situations, I think larger sites use more selective approach on what to cache (I vaguely remember reading wordpress.com does that with batcache);

serving gzipped content is widely accepted practice, but it trades bandwidth for CPU and can cause processor load issues accordingly.

Really caching and performance is giant topic. I barely scratched it myself.

My personal advices:

Read the docs. Everyone using caching plugins needs to have at least basic understanding of what is going on.

Don't be greedy with advanced toys. Some options are simply put for big league, shared hosting or even VPS with slow disk can suffer from them.

Ask your hosting. If they have any kind of decent support in place they will be able to come up with at least basic recommendation of what to use so it plays well with other stuff on server and doesn't cause overload.

W3 Total Cache is an awesome plugin (I use it), but if it is having issues or is way over your head, give WP Super Cache a try.

The latest version of WP Super Cache is quite easy to use and has a very simple interface.

Should only take a few minutes to setup. There shouldn't be any negative impact to general WordPress functions. If you have some odd/unique/custom plugins, there might be some rare issues with caching.