Subject: Re: Benchmark Results for CL's onWin32
From: Erik Naggum <erik@naggum.no>
Date: 1999/12/02
Newsgroups: comp.lang.lisp
Message-ID: <3153166270117604@naggum.no>
* "Martin Simmons" <martin@harlequin.co.uk>
| For best performance, you should also turn off all debugging support.
all of these benchmarks seem to be testing fairly unusual conditions.
the arguments we see here for "tuning" seem to me to imply that we have
to chuck all the Lispy features we can think of to get a comparison. I
think this is completely bogus. I'm not going to trust _myself_ enough
to run my code with the kinds of optimization settings people advocate
for these benchmarks. I _want_ a lot of dynamism, debugging support,
etc, and I'm not usually ready to run my code in production without these
things, except in particular functions that I have determined to need it,
and which have been specially written to be blindingly fast. however, in
the latter case, I'd be pretty damn stupid if I thought I could have 100%
portable ANSI Common Lisp code _and_ super-duper maximal performance at
the same time. _sheesh_, guys. if I had such needs as you imply with
these declarations for "best performance", I would even hack the compiler
output and look for instruction scheduling opportunities myself. using a
special function to turn off debugging, for instance, is just _one_ such
measure to squeeze the bejeezus out of the system, and we're already
outside of the portable ANSI Common Lisp domain.
so in my view, you either want super-fast code and profile it and get the
optimal results with hand-tweaked code that may not work anywhere else,
or you use fully portable code under _normal_ condtions. anything else
is very close to cheating, rendering the benchmark results evidence of
how good you are at cheating at random in each implementation with no way
of comparing your skills and then _nothing_ else. if you let instead
consciously and openly do your extreme best to write fast code for each
implementation, and _then_ compare the results, you at the very least get
comparable results for comparable _people_ (i.e., experts).
I suggest such benchmarks limit themselves to the narrowest possible type
declarations, the highest SPEED and the lowest DEBUG and SAFETY, and
either full local tweaking or no local tweaking. if a declaration isn't
enough, that's a factor that you need to take into account when and if
you engage in _full_ local tweaking, and not at all until then.
#:Erik