In playing with and comparing the various answers to my question here, I tried applying Timing[] to see how fast things ran, but it seemed that some of the answers (but not all of them) were a lot faster when run for a second time. How can I make sure that there isn't some kind of caching of intermediate results throwing off the Timing[]?

Not your question, but perhaps it is better to use AbsoluteTiming for "benchmarking", Timing does strange stuff. For example it has problems with multiple CPUs.
–
user21Feb 1 '12 at 17:58

@ruebenko: That sounds like a good idea—it's often very useful to answer the question you think I should have been asking (or try to address the underlying or larger issue), rather than just focusing on what I actually asked.
–
IsaacFeb 1 '12 at 20:17

It's hard to say whether playing with these will in fact give more reliable results, I could well imagine that some stuff just gets very slow. But it's probably worth a try.

A completeley different approach that I have used before is to ensure that there is nothing that can be cached, which often but not always is possible. One way is to achieve this is to use some randomness in the code you want to test, see e.g. my answer to this question.

If I were to change these, will they be reset to the defaults when I quit and relaunch Mathematica?
–
IsaacFeb 1 '12 at 20:25

I don't think so, but I think you can reset them by setting them to Automatic. Nevertheless you problably want to store the defaults before starting to play with these. I should probably also cite the documentation of SetSystemOptions: System options specify parameters relevant to the internal operation of Mathematica. They should not normally be modified except under specific direction. If they are modified, Mathematica may not operate correctly.
–
Albert ReteyFeb 1 '12 at 20:33

Oh, and of course I'm by no means entitled to give you those specific directions :-)
–
Albert ReteyFeb 1 '12 at 20:47

@Isaac Yes if you call SetSystemOptions["CacheOptions" -> "CacheKeyMaxBytes" -> 0]; Once you quit the kernel it will reset to 1000000.
–
mmorrisMar 12 '12 at 22:58

It is more reliable to just interleave ClearSystemCache[] between calls to whatever you are timing. Changing these system options is quite safe. The problem is that some functions, e.g. Together and Integrate, use intermediate caching to speed themselves (not just later calls to same function). So you could get degraded performance that would not reflect usual behavior if caching is disabled for intermediate steps.
–
Daniel LichtblauMar 13 '12 at 15:41

When experimenting with Mathematica's caching, sometimes it can be clearer to look at memory usage rather than timing. When looking at individual functions, Mathematica is able to measure memory much more accurately than timing. The are a number of reasons why timing accuracy is off, such as the system clock resolution and fluctuations in the system use while testing. For more of an example check out the answer to this question.

When using ClearSystemCache, it has to be continually called in benchmarking studies. Lets look at the following example:

One call

Quit[]; (* Has to be in a cell by itself or the rest will not be evaluated. *)
m = MemoryInUse[];
N[Pi, 10^5];
MemoryInUse[] - m

Results:

42368

Iterating over the call 10 times. Not clearing the cache. FYI I am using map to collect the values for me.

I have not been able to explain the difference in memory used, 42368 vs 42304 vs 42072. Maybe Map is caching something? Note, that clearing / turning off caching may not effect all functions. Take Factorial (!) There must be some other caching going on. Neither ClearSystemCache nor SetSystemOptions has an effect. Both:

"FYI I am using map to collect the values for me." -- why use Map that way when you could use Array or Table? (+1 on your answer, by the way)
–
Mr.Wizard♦Mar 13 '12 at 7:42

@Mr.Wizard No particular reason, other than I came across Map and have not much exposure to Array or Table. I have found my work in Mathematica a bit ... evolutionary. ;) I'll take a look. Thanks
–
mmorrisMar 13 '12 at 13:57

I see. I didn't mean to criticize, though I do think those are more standard in this application. Many people here have their own methods and tricks and I have learned a lot from them.
–
Mr.Wizard♦Mar 13 '12 at 14:01

@Mr.Wizard I did not take it as criticism. I appreciate the advice.
–
mmorrisMar 13 '12 at 14:43

An example of how my work has evolved: Summing the items of the list = {1, 2, 3, 4}. Background I was looking for a one argument function like Min, Max, Mean, ... and did not have much initial success with help. I started with: sum = 0; i = 0; While[i++ < Length[list], sum += Part[list, i]]; sum which evolved to: Fold[Plus, 0, list] followed by: Sum[i, {i, list}] and finally: Total[list]. I used "finally" with hesitation, as there is always a chance that there is something more efficient. ;)
–
mmorrisMar 13 '12 at 15:30

Mathematica is a registered trademark of Wolfram Research, Inc. While the mark is used herein with the limited permission of Wolfram Research, Stack Exchange and this site disclaim all affiliation therewith.