[Sbcl-devel] core image predictability

Hi,
when using the random number generator initialized from a predictable
source, the random numbers will stay predictable. As will the object
ID numbers of created objects when executing exactly the same forms
in the same order from the same core image. But when saving core
images using #'save-lisp-and-die, the core images always differ even
with only predictable computations done. Where does the differing
information come from? Would it be difficult to create a version of
#'save-lisp-and-die that creates predictable core images?
Best regards,
Isidor

Thread view

Hi,
when using the random number generator initialized from a predictable
source, the random numbers will stay predictable. As will the object
ID numbers of created objects when executing exactly the same forms
in the same order from the same core image. But when saving core
images using #'save-lisp-and-die, the core images always differ even
with only predictable computations done. Where does the differing
information come from? Would it be difficult to create a version of
#'save-lisp-and-die that creates predictable core images?
Best regards,
Isidor

AFAIU, if you want "predictability" in you random numbers, you'd
better save the random state and re-read it at startup.
Cheers
--
Marco
On Apr 19, 2008, at 13:52 , Isidor Zeuner wrote:
> Hi,
>
> when using the random number generator initialized from a predictable
> source, the random numbers will stay predictable. As will the object
> ID numbers of created objects when executing exactly the same forms
> in the same order from the same core image. But when saving core
> images using #'save-lisp-and-die, the core images always differ even
> with only predictable computations done. Where does the differing
> information come from? Would it be difficult to create a version of
> #'save-lisp-and-die that creates predictable core images?
>
> Best regards,
>
> Isidor
>
> ----------------------------------------------------------------------
> ---
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
> Don't miss this year's exciting event. There's still time to save
> $100.
> Use priority code J8TL2D2.
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://
> java.sun.com/javaone
> _______________________________________________
> Sbcl-devel mailing list
> Sbcl-devel@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-devel
--
Marco Antoniotti

On Sat, Apr 19, 2008 at 12:52 PM, Isidor Zeuner <sbcl@...> wrote:
> Hi,
>
> when using the random number generator initialized from a predictable
> source, the random numbers will stay predictable. As will the object
> ID numbers of created objects when executing exactly the same forms
> in the same order from the same core image. But when saving core
> images using #'save-lisp-and-die, the core images always differ even
> with only predictable computations done. Where does the differing
> information come from? Would it be difficult to create a version of
> #'save-lisp-and-die that creates predictable core images?
If your inputs are fully deterministic (same user running on the same
host, in the same directory), and there are no hidden sources of
indeterminism (like allocation patterns changing slightly when
multiple threads are running, depending on the order the OS scheduler
happens to schedule them), the only things I can think of that should
change are time-related.
If that is indeed the only source of indeterminism for you
(predictable inputs, only a single thread running), and having
bit-identical cores is important (why?), then it should not be too
hard to force the timestamps to a known values.
Cheers,
-- Nikodemus

> If that is indeed the only source of indeterminism for you
> (predictable inputs, only a single thread running), and having
> bit-identical cores is important (why?), then it should not be too
> hard to force the timestamps to a known values.
>
I didn't have the time to further investigate this, but will
eventually do later. Are there any particular places to look for these
sources of time-related information?
The reason for trying this is to use the lisp images in a more
instrumentative way. The idea was to provide a similar facility like
compiler caches used for C compilers. These regard the preprocessed C
source as the input to a function and the object file as its
output. Then this function is memoized, so multiple compilations of
the same source code will run faster. To translate this to Common Lisp
systems with core saving, I regard the Lisp REPL invocation with a
call to #'save-lisp-and-die at its end as a function taking a lisp
core and some lisp forms as its input and providing a new core as its
output. This way, I could memoize core creations, leading to speedup
when creating cores in an incremental way. With unpredictable cores,
however, there are core creations unneccessarily lost.
Best regards,
Isidor

It's interesting that you should be doing something like that (a build
system that caches images), because it's also one of my projects for
the summer. Could we collaborate?
Unless you always compile a whole system at a time, one of our issues
was also the fact that we needed a way to reproduce the compile-time
effects of compiling a file without spending the time re-compiling it.
Juho Snellman has produced a patch to SBCL to produce a .cfasl file at
the same time as a .fasl.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
No clever quote today.
2008/4/28 Isidor Zeuner <sbcl@...>:
> > If that is indeed the only source of indeterminism for you
> > (predictable inputs, only a single thread running), and having
> > bit-identical cores is important (why?), then it should not be too
> > hard to force the timestamps to a known values.
> >
>
> I didn't have the time to further investigate this, but will
> eventually do later. Are there any particular places to look for these
> sources of time-related information?
>
> The reason for trying this is to use the lisp images in a more
> instrumentative way. The idea was to provide a similar facility like
> compiler caches used for C compilers. These regard the preprocessed C
> source as the input to a function and the object file as its
> output. Then this function is memoized, so multiple compilations of
> the same source code will run faster. To translate this to Common Lisp
> systems with core saving, I regard the Lisp REPL invocation with a
> call to #'save-lisp-and-die at its end as a function taking a lisp
> core and some lisp forms as its input and providing a new core as its
> output. This way, I could memoize core creations, leading to speedup
> when creating cores in an incremental way. With unpredictable cores,
> however, there are core creations unneccessarily lost.
>
> Best regards,
>
> Isidor
>
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
> Don't miss this year's exciting event. There's still time to save $100.
> Use priority code J8TL2D2.
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> Sbcl-devel mailing list
> Sbcl-devel@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-devel

"Isidor Zeuner" <sbcl@...> writes:
> I didn't have the time to further investigate this, but will
> eventually do later. Are there any particular places to look for these
> sources of time-related information?
>
> The reason for trying this is to use the lisp images in a more
> instrumentative way. The idea was to provide a similar facility like
> compiler caches used for C compilers. These regard the preprocessed C
> source as the input to a function and the object file as its
> output. Then this function is memoized, so multiple compilations of
> the same source code will run faster. To translate this to Common Lisp
> systems with core saving, I regard the Lisp REPL invocation with a
> call to #'save-lisp-and-die at its end as a function taking a lisp
> core and some lisp forms as its input and providing a new core as its
> output. This way, I could memoize core creations, leading to speedup
> when creating cores in an incremental way. With unpredictable cores,
> however, there are core creations unneccessarily lost.
The first thing that comes to mind is SB!C::SOURCE-INFO-START-TIME.
But more generally getting you'll never be able to get absolutely
repeatable core files, due to the GC being conservative on the stack
and the registers. On one run some stack cell might contain a
time-based unboxed value that looks like a valid pointer to a lisp
object, causing a gc to run slightly differently, which will than
quickly snowball into more and more changes.
--
Juho Snellman