Hi Mark Jason. My name is Josh Engelman; I'm a software developer at
D.E. Shaw & Co. We've noticed a problem with Memoize.pm when used with
multiple threads in perl 5.8.6. I'm not sure if you're still
maintaining this module; if there's someone else you think we should
contact, please let me know.
The problem seems to be that function addresses change in the new
thread, but the %memotable and %revmemotable keys still refer to the old
function addresses. This means that any attempt to use a memoized
function from a second thread raises an exception that kills that
thread. Here's a very simple demonstration:
$ perl -e '
use threads;
use Memoize;
Memoize::memoize("foo");
sub foo { $_[0] }
my $t1 = threads->new(\&foo, 1);
printf "%d\n", $t1->join();
'
thread failed to start: Anonymous function called in forbidden
scalar context; faulting at -e line 6
0
This is unfortunate, because it means that any use of a memoized
function in *any* perl module makes that module unsafe to use in
multi-threaded programs. Since Memoize.pm is part of the core perl
distribution, this has far-reaching ramifications.
I haven't been able to come up with a good way to make Memoize
generically thread-safe -- doing so requires re-engineering of the way
memotables work, or rebuilding them when the new thread is created.
However, a very small patch will allow memoized functions to continue
working in subsequent threads. Basically, the idea is to pass the
serialized version of $cref to the _memoizer(), rather than passing the
reference itself. The reference itself will have a different serialized
form in subsequent threads, but we want to continue using the same
%memotable key.
Here's the patch diff:
diff -c /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm
Memoize.pm
*** /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm Fri Jul
12 13:16:00 2002
--- Memoize.pm Mon Sep 25 13:21:24 2006
***************
*** 72,78 ****
# goto version everywhere.
my $wrapper =
$Config{usethreads}
! ? eval "sub $proto { &_memoizer(\$cref, \@_); }"
: eval "sub $proto { unshift \@_, \$cref; goto
&_memoizer; }";
my $normalizer = $options{NORMALIZER};
--- 72,78 ----
# goto version everywhere.
my $wrapper =
$Config{usethreads}
! ? eval "sub $proto { &_memoizer('$cref', \@_); }"
: eval "sub $proto { unshift \@_, \$cref; goto
&_memoizer; }";
my $normalizer = $options{NORMALIZER};
Note that this solution will not fix unmemoize() in subsequent threads,
nor address any complications caused by tied caches. But I believe it
fixes the most common case.
I've attached the full patched module along with a slightly more
extensive test script. Note that I've tested both the problem and the
solution with perl 5.8.6 on Solaris 10 x86, Solaris 10 sparc, Windows
XP, <<Memoize.pm>> a <<break_memoize.pl>> nd MacOS 10.4. Please let me
know if I can provide any further information or assistance. Thx.
-josh
===
Josh Engelman
Vice President, D.E. Shaw & Co.
Ph. 212.478.0614
Fax 212.845.1614