Fiddler works by storing the entire request and response in memory. If you are performing a huge download (hundreds of megabytes) it's possible that Fiddler cannot find a free memory block large enough to hold the entire contiguous response, and hence you'll run into this "out of memory" problem. It's also possible that if you have thousands of sessions in the Fiddler session list, even a relatively small memory block will not be available to store a response a few megabytes in size. You can reduce the incidence of this problem by clearing the Web Sessions list (CTRL+X) or configuring it to automatically trim to the most recent two hundred sessions (Click the Filters tab, and click the "Keep only the most recent sessions" option at the bottom).

Update: Fiddler2 now supports running on 64bit computers. If you're on a 64-bit machine, you'll never hit a problem.

If you're on a 32-bit machine, you can avoid out-of-memory errors when downloading huge files by adding the following code inside the OnPeekAtResponseHeaders function inside Rules > Customize Rules. The line in red will cause Fiddler not to keep a copy of the large file: