William Harold Newman wrote:
>
> On Tue, Jul 24, 2001 at 06:07:18PM -0400, Sam Steingold wrote:
> > The current CLISP development sources support MAKE-LOAD-FORM, which is
> > alleged to be the only missing feature preventing CMUCL from being
> ^^^^^
> > bootstrapped with CLISP.
>
> As someone (Eric Marsden, I think) pointed out, SBCL is the system
> whose bootstrapping under CLISP was failing because MAKE-LOAD-FORM
> doesn't exist. I don't know whether it's the only problem, but it's
> enough of a problem that bootstrapping didn't get very far without it.
It's not the only problem. I tried it out some days ago, and had to
patch SBCL to work around some - I guess - clisp specific problems.
The problems arise quite early in the compilation process, e.g. in
compiling the cross-compiler:
In clisp, for example, DOCUMENTATION doesn't work for package objects.
(documentation (find-package :cl-user) t) => error!
There is a problem with RENAME-FILE-A-LA-UNIX in src/cold/shared.lisp,
where we create a temporary file, but clisp can't rename another file
to the name of the temporary file, so a simple workaround is to delete
this temp-file in the clisp-case.
Some of Bill's workaround to bootstrap with clisp are not necessary
anymore
- cf. comments and #+clisp stuff in src/cold/ansify.lisp and
src/cold/shared.lisp
These can be deleted.
There is also a problem with the reader-macro setup (and that's where I
stopped)
...
; Loading file obj/from-host/src/code/backq.lisp-obj ...
;; Loading of file obj/from-host/src/code/backq.lisp-obj is finished.
Compiling file
/home/ma/data/lisp/sbcl/sbcl-061248-test/src/code/defsetfs.lisp
...
*** - READ from
#<INPUT BUFFERED FILE-STREAM CHARACTER
#P"/home/ma/data/lisp/sbcl/sbcl-061248-test/src/code/defsetfs.lisp"
@21>: macro character definition for #\, may not return 2 values, only
one value.
So, I hope this helps a bit, if someone wants to continue that quest.
:)
Cheers,
Martin
--
Martin Atzmueller <martin@...>

I've now dealt with
* the half-dozen or so patches that I received from Martin Atzmueller
before I left for the American Go AssociationCongress
* the new patch FIX-CORE-SOURCE-INFO patch from Martin Atzmueller
* the FILL-POINTER-OUCH patch from Tim Moore on cmucl-imp
I also fixed some old bugs (number 26, number 55, non-ANSIness of
treating inferred function type as a declaration, and need for two
Ctrl-D to terminate input).
Thank you, Martin! (and Tim Moore too, if you're out there)
Since I didn't have Internet access at the AGA Congress, I did
everything in a temporary CVS repository on my laptop. Sometime real
soon, hopefully later today, I plan to do the necessary diffs and
commits and whatnot to update SourceForge CVS, bringing it up to
version sbcl-0.6.12.58 or so in one fell swoop.
Probably after a little more messing with that version (especially
seeing whether it's possible to bootstrap with CLISP), I'll release it
as sbcl-0.6.13. After sbcl-0.6.13, I intend to start making the
various more-radical changes that I'd deferred until sbcl-0.7.0.
I'm not sure what the CVS tree structure will look like at that point.
I'd like to leave the possibility of releasing sbcl-0.6.14 later, with
no radical changes, only minor bugfixes. Maybe that will become
stable_0_6_branch or something.
I haven't decided what the version numbering on the main CVS branch
will look like in the transition from sbcl-0.6.13 to sbcl-0.7.0. Maybe
it will go "0.6.13", "0.pre7.0", "0.pre7.1", .. "0.pre7.33", "0.7.0".
--
William Harold Newman <william.newman@...>
"The King with half the East at heel is marched from lands of morning;
His fighters drink the rivers up, their shafts benight the air,
And he that stays will die for naught, and home there's no returning.
The Spartans on the sea-wet rock sat down and combed their hair."
-- A.E. Housman
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

On Mon, Jul 30, 2001 at 03:39:43PM +0200, Martin Atzmueller wrote:
> Ernst Jeschek wrote:
>
> > I've tried to compile net-sbcl-sockets 0.2.1 with sbcl 0.6.12.48
> > on OpenBSD 2.9.
> >
> > I ran into the following error:
> >
> > | debugger invoked on condition of type SIMPLE-ERROR:
> > | unknown foreign symbol: "getprotobyname"
> >
> > Has someone successfully tried this?
> net.sbcl.sockets has been built on linux, and as far as I know,
> on FreeBSD, sucessfully.
>
> Could you provide some more details about the compilation process?
> (I don't have OpenBSD, so I cannot try it myself)
The foreign symbol stuff on OpenBSD isn't very mature, since I don't
use the foreign symbol stuff myself. It's also somewhat different from
Linux and FreeBSD since it's still built statically (as per
"OS_LINK_FLAGS = -static" in "Config.x86-openbsd"). I have an old note
from Raymond Wiker suggesting how I might be able to build it with
dynamic linking, but I still haven't gotten around to trying it.
--
William Harold Newman <william.newman@...>
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

On Tue, Jul 24, 2001 at 06:07:18PM -0400, Sam Steingold wrote:
> The current CLISP development sources support MAKE-LOAD-FORM, which is
> alleged to be the only missing feature preventing CMUCL from being
^^^^^
> bootstrapped with CLISP.
As someone (Eric Marsden, I think) pointed out, SBCL is the system
whose bootstrapping under CLISP was failing because MAKE-LOAD-FORM
doesn't exist. I don't know whether it's the only problem, but it's
enough of a problem that bootstrapping didn't get very far without it.
> I would appreciate if someone tried it out.
I just returned from a trip and have several other things I have to
do, but I expect I'll get to it in a few days. (I'm very interested in
being able to bootstrap from CLISP, since CLISP is free and can itself
be bootstrapped from the ubiquitous GCC.)
--
William Harold Newman <william.newman@...>
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

Martin Atzmueller wrote:
> "Brown, Robert E (FICC)" wrote:
>
> > I tried building the HEAD revision of sbcl using both sbcl and cmucl. I got
> > the same error each time. I'll try again. Maybe I need a newer sbcl to
> > build the latest sbcl?
[...]
> As I wrote before, using cmucl-18c won't probably work.
> One problem again - in using cmucl, as I see it just now, is that it
> causes "warnings" about some "undefined" functions, when building
> the cross-compiler.
> These functions are compiled later, but are being referenced in code
> to be compiled previously.
> And since sbcl takes these warnings seriously, it won't proceed in
> the compilation process.
Well, I think I have to rectify this remark: the "warnings about
undefined functions" are not the reason, why CMUCL won't compile
current CVS-sbcl (0.6.12.48).
[At least, if the functions are defined later ;),
and this seems to work!]
The reason is another warning:
Actually, a bug has crept into SBCL (I think it was introduced in
0.6.12.45):
FIX-CORE-SOURCE-INFO (in src/compiler/generic/core.lisp)
should take three arguments, but is defined to take two,
and it's called both with two and three arguments.
So, CMUCL issues a WARNING, and that blows up compilation, at least
that's what happens in cmucl-18c.
The attached patch corrects for this, and now cmucl-18c happily builds
SBCL again.
--
Martin Atzmueller <martin@...>

"Brown, Robert E (FICC)" wrote:
> I tried building the HEAD revision of sbcl using both sbcl and cmucl. I got
> the same error each time. I'll try again. Maybe I need a newer sbcl to
> build the latest sbcl?
Well, probably. sbcl-0.6.8 is old, and there were some important
bugfixes since then, concerning the eval-when handling, too.
And the "eval-when"-stuff is probably the root of your problem.
Well, I had sent a patch to hack around this, since the problem came
up using CMUCL 18b, too, but there have been a lot of changes in sbcl,
lately. And few people seem to compile SBCL with CMUCL these days. :-|
As I wrote before, using cmucl-18c won't probably work.
One problem again - in using cmucl, as I see it just now, is that it
causes "warnings" about some "undefined" functions, when building
the cross-compiler.
These functions are compiled later, but are being referenced in code
to be compiled previously.
And since sbcl takes these warnings seriously, it won't proceed in
the compilation process.
So, I'd recommend using sbcl-0.6.12, to build from the current CVS
sources
(the last version I used to rebuild was 0.6.12.33, so I guess it'll
work ...)
Cheers,
Martin
--
Martin Atzmueller <martin@...>

Yes, it was late at night ... forgot to adjust the exponent.
Sorry about not including version information. I am using sbcl 0.6.8 and
some version of cmucl 18b.
I tried building the HEAD revision of sbcl using both sbcl and cmucl. I got
the same error each time. I'll try again. Maybe I need a newer sbcl to
build the latest sbcl?
Thanks very much for your reply.
bob
-----Original Message-----
From: Martin Atzmueller [mailto:martin@...]
Sent: Monday, July 23, 2001 9:47 AM
To: Brown, Robert E (FICC)
Cc: sbcl-devel@...
Subject: Re: [Sbcl-devel] two bugs?
(Sorry, I hit "send" way to early in the previous mail)
"Robert E. Brown" wrote:
>
> I just reported these bugs to a CMU CL mailing list, but they apply to
sbcl
> as well, so I'm reporting them here too. I would have taken a stab at
> fixing these myself, but I'm having trouble building the HEAD revision of
> sbcl from the CVS repository. Transcript of the breakage is attached
below.
[...]
> I think I've found an off-by-one bug in the code to print floating point
> numbers. Evaluating (format t "~,9,,3,,,EE" 0.000000001) produces
> 1000.0000000e-12E instead of 100.00000000e-12E.
I confirm that (in SBCL-0.6.12.48), but it should probably be
100.0000000E-11E.
:)
> Also, I think there's a bug in the handling of t or * in cons type
> specifiers. Both of the following evaluate to (values nil nil). I expect
> (values t t).
>
> (subtypep '(cons (member foo) t) '(cons (member foo) *))
> (subtypep '(cons (member foo) *) '(cons (member foo) t))
Both of these do evaluate to (values t t) in SBCL-0.6.12.48.
Which version of SBCL are you using?
(Onto the breakage):
> ====================
>
> obj/from-host/src/compiler/generic/early-objdef.lisp-obj-tmp written
> compilation finished in 0:00:00
> creating directory: obj/from-host/src/compiler/target/
> compiling file "/arm/software/cvs-trees/sbcl/src/compiler/x86/parms.lisp"
> (written 11 JUN 2001 09:44:07 AM):
[...]
> debugger invoked on SB-DEBUG:*DEBUG-CONDITION* of type UNBOUND-VARIABLE:
> error in SB-KERNEL::UNBOUND-SYMBOL-ERROR-HANDLER: The variable
> #:EXPR-TMP-11 is unbound.
> restarts:
> 0: [ABORT] Return to toplevel.
> Within the debugger, you can type HELP for help.
> (EVAL #:EXPR-TMP-11)
> 0]
This looks like a bug in the interaction with CMUCL again.
Your output looks like you are using CMUCL.
src/code/primordial-extensions
contains a hack that made it possible to build SBCL with CMUCL, because
there were problems with the DEFCONSTANT-EQX macro. CMUCL obviously has
a bug in EVAL-WHEN handling, and code src/compiler/x86/parms.lisp
has some DEFCONSTANT-EQX stuff in it, that probably causes your
problems.
You should probably use SBCL to build from CVS, which worked for me.
I've tried to rebuild the current HEAD branch with CMUCL (18c), but if
failed when compiling/loading src/compiler/main.lisp.
Need to look into that.
Has anybody successfully built SBCL with CMUCL recently?
Cheers,
Martin
--
Martin Atzmueller <martin@...>

(Sorry, I hit "send" way to early in the previous mail)
"Robert E. Brown" wrote:
>
> I just reported these bugs to a CMU CL mailing list, but they apply to sbcl
> as well, so I'm reporting them here too. I would have taken a stab at
> fixing these myself, but I'm having trouble building the HEAD revision of
> sbcl from the CVS repository. Transcript of the breakage is attached below.
[...]
> I think I've found an off-by-one bug in the code to print floating point
> numbers. Evaluating (format t "~,9,,3,,,EE" 0.000000001) produces
> 1000.0000000e-12E instead of 100.00000000e-12E.
I confirm that (in SBCL-0.6.12.48), but it should probably be
100.0000000E-11E.
:)
> Also, I think there's a bug in the handling of t or * in cons type
> specifiers. Both of the following evaluate to (values nil nil). I expect
> (values t t).
>
> (subtypep '(cons (member foo) t) '(cons (member foo) *))
> (subtypep '(cons (member foo) *) '(cons (member foo) t))
Both of these do evaluate to (values t t) in SBCL-0.6.12.48.
Which version of SBCL are you using?
(Onto the breakage):
> ====================
>
> obj/from-host/src/compiler/generic/early-objdef.lisp-obj-tmp written
> compilation finished in 0:00:00
> creating directory: obj/from-host/src/compiler/target/
> compiling file "/arm/software/cvs-trees/sbcl/src/compiler/x86/parms.lisp"
> (written 11 JUN 2001 09:44:07 AM):
[...]
> debugger invoked on SB-DEBUG:*DEBUG-CONDITION* of type UNBOUND-VARIABLE:
> error in SB-KERNEL::UNBOUND-SYMBOL-ERROR-HANDLER: The variable
> #:EXPR-TMP-11 is unbound.
> restarts:
> 0: [ABORT] Return to toplevel.
> Within the debugger, you can type HELP for help.
> (EVAL #:EXPR-TMP-11)
> 0]
This looks like a bug in the interaction with CMUCL again.
Your output looks like you are using CMUCL.
src/code/primordial-extensions
contains a hack that made it possible to build SBCL with CMUCL, because
there were problems with the DEFCONSTANT-EQX macro. CMUCL obviously has
a bug in EVAL-WHEN handling, and code src/compiler/x86/parms.lisp
has some DEFCONSTANT-EQX stuff in it, that probably causes your
problems.
You should probably use SBCL to build from CVS, which worked for me.
I've tried to rebuild the current HEAD branch with CMUCL (18c), but if
failed when compiling/loading src/compiler/main.lisp.
Need to look into that.
Has anybody successfully built SBCL with CMUCL recently?
Cheers,
Martin
--
Martin Atzmueller <martin@...>

William Harold Newman wrote:
>
> On Mon, Jul 16, 2001 at 03:27:59PM +0200, Martin Atzmueller wrote:
> > I've ported the "irrat" patch from cmucl, based on a
> > patch by Ray Toy on cmucl-imp 2001-04-12:
> [and two other patches]
>
> Thank you, for this and the other two patches as well. I'm afraid I'm
> not going to get to any of them for two weeks, since I'm in a frenzy
Ok.
> of preparations to leave town on July 20 and not get back until July
> 29, during which trip I'll do lots of things related to the game of Go
Well, good luck, then!
Shall the force (of SBCL) be with you. ;)
--
Martin Atzmueller <martin@...>

On Mon, Jul 16, 2001 at 03:27:59PM +0200, Martin Atzmueller wrote:
> I've ported the "irrat" patch from cmucl, based on a
> patch by Ray Toy on cmucl-imp 2001-04-12:
[and two other patches]
Thank you, for this and the other two patches as well. I'm afraid I'm
not going to get to any of them for two weeks, since I'm in a frenzy
of preparations to leave town on July 20 and not get back until July
29, during which trip I'll do lots of things related to the game of Go
-- like watching my Go-playing program get utterly crushed, alas --
and do very little related to the Internet, including SBCL. (But then
once I get back, I plan to do quite a bit of pent up work on SBCL,
probably including 0.7.0 sometime reasonably soon.)
--
William Harold Newman <william.newman@...>
"The King with half the East at heel is marched from lands of morning;
His fighters drink the rivers up, their shafts benight the air,
And he that stays will die for naught, and home there's no returning.
The Spartans on the sea-wet rock sat down and combed their hair."
-- A.E. Housman
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

Attached is a patch ported from cmucl-imp 2001-03-19, by Ray Toy,
e.g.
"...
Here is a patch that implements defoptimizers for array-element-type
and coerce. I think array-element-type does the right thing in all
cases. The optimizer for coerce isn't quite as smart as it could be
but it solves the case that Paul was asking about. This could use
more work.
Ray
Changelog:
compiler/srctran.lisp:
o Add defoptimizers for array-element-type and coerce.
"
--
I've attached some demo-code to see the effects of that, too.
Just do a DESCRIBE on these functions, and the return values
should speak for themselves.
Cheers,
Martin
--
Martin Atzmueller <martin@...>

(sorry for the duplicate mail, but this one should have the attachment!)
I've looked into the wait-for-floating-point-exceptions issue
in handler-case, brought up on cmucl-imp around 2001-03-28.
This was also on the TODO list for pre-0.7-SBCL (referring to
Bill's mail on 2001-05-09).
There was a problem in CMUCL, that caused a floating-point-exception,
and for that the patch for handler-case was made (more onto that,
below).
Attached is a patch that has the float-wait instruction in the
handler-case macro, ported from CMUCL.
It turned out, that CMUCL was using another version of the macro,
that was commented out in the CMUCL-sources, but was reinstated
around 2000-07. A comment in the CVS logs for SBCL (sbcl-0.6.7.2),
said that the older-version of handler-case was deleted.
Hmm, I'm not sure, which one the older one was ... ;)
The new code looks a bit more "modern" and cleaner, since it uses
a TAGBODY for directing the control-flow, but that's merely a
matter of style.
Well, more importantly, since the original code to trigger the bug,
given by Christophe Rhodes, does yield no error in SBCL, neither
with the old/new handler-case: I'm not sure, if we should include
the patch, but we could do it as CMUCL did, with the old code commented
out. :-|
The patch may still be relevant for SBCL, but I haven't found a way
to trigger the bug, yet.
Here are some important pieces from cmucl-imp for that matter:
The code that triggered the problem:
* (defun make-single-float (bits)
(if (zerop bits) ; IEEE float special case
0.0
(let ((sign (ecase (ldb (byte 1 31) bits)
(0 1.0)
(1 -1.0)))
(expt (- (ldb (byte 8 23) bits) 127))
(mant (* (logior (ldb (byte 23 0) bits)
(ash 1 23))
(expt 0.5 23))))
(* sign (expt 2.0 expt) mant))))
MAKE-SINGLE-FLOAT
* (make-single-float 1)
5.8774716e-39
* (compile 'make-single-float)
Compiling LAMBDA (BITS):
Arithmetic error FLOATING-POINT-OVERFLOW signalled.
--
Subject: Re: Compiler bug?
From: Raymond Toy <toy@...>
Date: 19 Apr 2001 10:49:06 -0400
[...]
Raymond> 1. The underlying problem is in safe-expt, despite what the
backtrace
Raymond> says. The computation is (safe-expt 2.0 128), which
causes an
Raymond> overflow.
Raymond> 2. The real cause of 1 is that handler-case in safe-expt
doesn't
Raymond> correctly handle the floating-point overflow.
Raymond> 3. The backtrace is totally wrong. TWO-ARG-= can't
possibly cause an
Raymond> overflow.
Figured out what the problem is. If you macroexpand the handler-case
and handler-bind in safe-expt, you get something like:
(LET ((CONDITIONS::*HANDLER-CLUSTERS*
(CONS
(LIST
(CONS 'ERROR
#'(LAMBDA (CONDITIONS::TEMP)
(DECLARE (IGNORE CONDITIONS::TEMP))
(GO #:G948))))
CONDITIONS::*HANDLER-CLUSTERS*)))
(MULTIPLE-VALUE-PROG1 (PROGN (RETURN-FROM #:G946 (EXPT X Y)))
(KERNEL:FLOAT-WAIT)))
Since FP exceptions aren't signaled until the *next* FP instruction,
the expt returns and bypasses the float-wait. I think handler-bind
needs the float-wait call.
The appended patch does this by adding a float-wait for
handler-case. With this change, FP errors should be handled at the
appropriate time.
[...]
--------
Cheers,
Martin
--
Martin Atzmueller <martin@...>

I've looked into the wait-for-floating-point-exceptions issue
in handler-case, brought up on cmucl-imp around 2001-03-28.
This was also on the TODO list for pre-0.7-SBCL (referring to
Bill's mail on 2001-05-09).
There was a problem in CMUCL, that caused a floating-point-exception,
and for that the patch for handler-case was made (more onto that,
below).
Attached is a patch that has the float-wait instruction in the
handler-case macro, ported from CMUCL.
It turned out, that CMUCL was using another version of the macro,
that was commented out in the CMUCL-sources, but was reinstated
around 2000-07. A comment in the CVS logs for SBCL (sbcl-0.6.7.2),
said that the older-version of handler-case was deleted.
Hmm, I'm not sure, which one the older one was ... ;)
The new code looks a bit more "modern" and cleaner, since it uses
a TAGBODY for directing the control-flow, but that's merely a
matter of style.
Well, more importantly, since the original code to trigger the bug,
given by Christophe Rhodes, does yield no error in SBCL, neither
with the old/new handler-case: I'm not sure, if we should include
the patch, but we could do it as CMUCL did, with the old code commented
out. :-|
The patch may still be relevant for SBCL, but I haven't found a way
to trigger the bug, yet.
Here are some important pieces from cmucl-imp for that matter:
The code that triggered the problem:
* (defun make-single-float (bits)
(if (zerop bits) ; IEEE float special case
0.0
(let ((sign (ecase (ldb (byte 1 31) bits)
(0 1.0)
(1 -1.0)))
(expt (- (ldb (byte 8 23) bits) 127))
(mant (* (logior (ldb (byte 23 0) bits)
(ash 1 23))
(expt 0.5 23))))
(* sign (expt 2.0 expt) mant))))
MAKE-SINGLE-FLOAT
* (make-single-float 1)
5.8774716e-39
* (compile 'make-single-float)
Compiling LAMBDA (BITS):
Arithmetic error FLOATING-POINT-OVERFLOW signalled.
--
Subject: Re: Compiler bug?
From: Raymond Toy <toy@...>
Date: 19 Apr 2001 10:49:06 -0400
[...]
Raymond> 1. The underlying problem is in safe-expt, despite what the
backtrace
Raymond> says. The computation is (safe-expt 2.0 128), which
causes an
Raymond> overflow.
Raymond> 2. The real cause of 1 is that handler-case in safe-expt
doesn't
Raymond> correctly handle the floating-point overflow.
Raymond> 3. The backtrace is totally wrong. TWO-ARG-= can't
possibly cause an
Raymond> overflow.
Figured out what the problem is. If you macroexpand the handler-case
and handler-bind in safe-expt, you get something like:
(LET ((CONDITIONS::*HANDLER-CLUSTERS*
(CONS
(LIST
(CONS 'ERROR
#'(LAMBDA (CONDITIONS::TEMP)
(DECLARE (IGNORE CONDITIONS::TEMP))
(GO #:G948))))
CONDITIONS::*HANDLER-CLUSTERS*)))
(MULTIPLE-VALUE-PROG1 (PROGN (RETURN-FROM #:G946 (EXPT X Y)))
(KERNEL:FLOAT-WAIT)))
Since FP exceptions aren't signaled until the *next* FP instruction,
the expt returns and bypasses the float-wait. I think handler-bind
needs the float-wait call.
The appended patch does this by adding a float-wait for
handler-case. With this change, FP errors should be handled at the
appropriate time.
[...]
--------
Cheers,
Martin
--
Martin Atzmueller <martin@...>

I've ported the "irrat" patch from cmucl, based on a
patch by Ray Toy on cmucl-imp 2001-04-12:
The original message was:
"Appended is a patch for irrat.lisp that
o Fixes the declaration bug in complex-log-scaled
o Removes the old special function routines
o Adds logb-finite to help optimize the use of logb
o Removes some unneeded declarations since the compiler is smarter now
than when this was originally written.
o Added inhibit-warnings to coerce-to-complex-type since their
unavoidable.
o The cores of some routines are compiled with speed 3 and space 0 to
get some maybe-inline routines inlined.
Some of the speed 3 stuff might be gratuitous, but I use complex
numbers everywhere in my work, so I want these routines to be
fast. :-)
There appears to be one small bug in complex-tanh where I needed to
declare some variables. Otherwise the compiler dies on an underflow
for some reason. Need to look into that."
---
Since some functions were rewritten in that process, I've made up
some code to test these.
The attached test-code could maybe be a starting point for
a "math-functions" test, or such.
Cheers,
Martin
--
Martin Atzmueller <martin@...>

On Sat, Jul 14, 2001 at 04:02:08PM +0100, Daniel Barlow wrote:
> William Harold Newman <william.newman@...> writes:
> sbcl.nm should be safe for anything in our own runtime or statically
> linked to it - in other words, for any symbol that it lists. But
> the lookup used by def-alien-variable will go off and try dlsym() if
> it can't find stuff statically.
>
> environ is - as far as I can determine using grep(1) - the only
> occurrence of def-alien-variable in the SBCL code at present,
> incidentally.
>
> > Incidentally, I can see that this could be a really annoying problem,
> > so if there's any difficulty making an elegant solution, I'd be
> > willing to merge a quick and dirty patch for the meantime, at least as
> > long as it didn't look too dangerous.
>
> For the moment, for the .deb, I'll add something to /etc/sbclrc: this
> is a sufficently ugly solution to goad me into doing something
> elegant, and easily reversible when the elegant thing does happen. I
> guess the right thing to do is go away and read
> *{before,after}-save-initializations* stuff, and see how to do it with
> that.
OK.
By the way, in case you or anyone else reading this weren't already
aware of it, someone has been writing to cmucl-imp@... about a
scheme for magically making references to external (and I think
dynamic) symbols work cleanly across saved/loaded cores. I haven't
been paying enough attention to remember how close it is to this
problem, or how close it is to complete, but it might be possible to
crib something from there instead of working it out from scratch.
--
William Harold Newman <william.newman@...>
The way men live is so far removed from the way they should live
that anyone who abandons what is for what should be pursues his
own downfall rather than his preservation. - Niccolo Machiavelli
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

William Harold Newman <william.newman@...> writes:
> My first impulse is to do something like (1), though
> (a) probably through the *BEFORE-SAVE-INITIALIZATIONS* /
> *AFTER-SAVE-INITIALIZATIONS* mechanism instead of a new
> "remember to tweak environ here" step in toplevel setup, and
> (b) ideally by making this happen automatically for all
> foreign symbols which need this treatment, instead of
> as a special hack just for environ, e.g. perhaps DEF-ALIEN-VARIABLE
> should automatically install hooks in *AFTER-SAVE-INITIALIZATIONS*
> to update the address it refers to
Ah, there's an *AFTER-SAVE-INITIALIZATIONS* mechanism. Yes, that
makes sense.
> But I don't know much about the problem. In particular, I don't know
> enough about the problem to guess what's required to make (b) happen.
> Is there something special about environ (like the way that errno is
> very special)? Or is this a general problem with any variable? Or any
> label? (including function addresses?) What kind of magic does C code
> use to avoid this problem?
It's just a normal variable. I expect, though haven't checked, that
function addresses have the same problem. C code resolves names to
addresses at startup time, instead of three months earlier at core
dump time, so it doesn't have the problem.
> I notice on OpenBSD that environ is in sbcl.nm. Perhaps what's going
> on is that it's in sbcl.nm in Linux too, and SBCL figures that it
It's not. I assume that OpenBSD is statically linked against libc.
> knows where environ is for all time, when in fact that's not a safe
> assumption? If so, does anyone know for which other labels this is not
> a safe assumption? Should we stop using sbcl.nm to get to variables?
sbcl.nm should be safe for anything in our own runtime or statically
linked to it - in other words, for any symbol that it lists. But
the lookup used by def-alien-variable will go off and try dlsym() if
it can't find stuff statically.
environ is - as far as I can determine using grep(1) - the only
occurrence of def-alien-variable in the SBCL code at present,
incidentally.
> Incidentally, I can see that this could be a really annoying problem,
> so if there's any difficulty making an elegant solution, I'd be
> willing to merge a quick and dirty patch for the meantime, at least as
> long as it didn't look too dangerous.
For the moment, for the .deb, I'll add something to /etc/sbclrc: this
is a sufficently ugly solution to goad me into doing something
elegant, and easily reversible when the elegant thing does happen. I
guess the right thing to do is go away and read
*{before,after}-save-initializations* stuff, and see how to do it with
that.
-dan
--
http://ww.telent.net/cliki/ - Link farm for free CL-on-Unix resources

On Sat, Jul 14, 2001 at 01:30:21AM +0100, Daniel Barlow wrote:
>
> For anyone who wasn't already aware, the cCLan (Comprehensive Common
> Lisp Archive Network) project periodically builds Debian binary
> packages of SBCL snapshot releases. What we've noticed now that
> people are trying to use them is that POSIX-ENVIRON fails when running
> an SBCL binary that was built using an even slightly different version
> of libc.
>
> * (posix-environ)
>
> debugger invoked on condition of type SIMPLE-ERROR:
> segmentation violation at #X163614A
>
> I'm guessing that dumping code containing the def-alien-variable form
> for SB-IMPL::ENVIRON is the culprit: I find it quite plausible that
> that would break if the address of "environ" changes. If I redefine
> the offending bits at the toplevel, it works fine thereafter
>
> * (in-package "SB-IMPL")
>
> #<PACKAGE "SB-IMPL">
> * (def-alien-variable "environ" (* c-string))
>
> #<SB-ALIEN-INTERNALS:HEAP-ALIEN-INFO
> (FOREIGN-SYMBOL-ADDRESS '"environ") (* C-STRING)>
> * (defun posix-environ ()
> "Return the Unix environment (\"man environ\") as a list of SIMPLE-STRINGs."
> (c-strings->string-list environ))
> STYLE-WARNING: redefining POSIX-ENVIRON in DEFUN
>
> POSIX-ENVIRON
> * (in-package :cl-user)
>
> #<PACKAGE "COMMON-LISP-USER">
> * (posix-environ)
>
> ("PWD=/home/dan" "CMUCLLIB=/home/dan" "LISPDIR=/home/dan/src/lisp"
> [etc]
>
>
> Fixes: either
>
> 1) define "environ" and POSIX-ENVIRON when the normal toplevel
> starts. Icky
>
> 2) change the code in run-foreign to do
>
> (def-alien-variable (environ "sbcl_environ") (* c-string))
>
> and add "extern char **sbcl_environ; " and something to initialize it
> in the C runtime.
>
> 3) something else? is there some kind of late-binding fixup we can
> do? my guess is "no", but it's too late at night
My first impulse is to do something like (1), though
(a) probably through the *BEFORE-SAVE-INITIALIZATIONS* /
*AFTER-SAVE-INITIALIZATIONS* mechanism instead of a new
"remember to tweak environ here" step in toplevel setup, and
(b) ideally by making this happen automatically for all
foreign symbols which need this treatment, instead of
as a special hack just for environ, e.g. perhaps DEF-ALIEN-VARIABLE
should automatically install hooks in *AFTER-SAVE-INITIALIZATIONS*
to update the address it refers to
Or if this is an environ-only problem, I might be inclined to do something
a little like (2), and a little like the way we deal with errno
already: define a read_environ() function in the C runtime (instead of
an initialized-once variable copy) and make the Lisp code call the
C-level read_environ() function when it wants to know the environment.
But I don't know much about the problem. In particular, I don't know
enough about the problem to guess what's required to make (b) happen.
Is there something special about environ (like the way that errno is
very special)? Or is this a general problem with any variable? Or any
label? (including function addresses?) What kind of magic does C code
use to avoid this problem?
I notice on OpenBSD that environ is in sbcl.nm. Perhaps what's going
on is that it's in sbcl.nm in Linux too, and SBCL figures that it
knows where environ is for all time, when in fact that's not a safe
assumption? If so, does anyone know for which other labels this is not
a safe assumption? Should we stop using sbcl.nm to get to variables?
Incidentally, I can see that this could be a really annoying problem,
so if there's any difficulty making an elegant solution, I'd be
willing to merge a quick and dirty patch for the meantime, at least as
long as it didn't look too dangerous.
--
William Harold Newman <william.newman@...>
The way men live is so far removed from the way they should live
that anyone who abandons what is for what should be pursues his
own downfall rather than his preservation. - Niccolo Machiavelli
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

For anyone who wasn't already aware, the cCLan (Comprehensive Common
Lisp Archive Network) project periodically builds Debian binary
packages of SBCL snapshot releases. What we've noticed now that
people are trying to use them is that POSIX-ENVIRON fails when running
an SBCL binary that was built using an even slightly different version
of libc.
* (posix-environ)
debugger invoked on condition of type SIMPLE-ERROR:
segmentation violation at #X163614A
I'm guessing that dumping code containing the def-alien-variable form
for SB-IMPL::ENVIRON is the culprit: I find it quite plausible that
that would break if the address of "environ" changes. If I redefine
the offending bits at the toplevel, it works fine thereafter
* (in-package "SB-IMPL")
#<PACKAGE "SB-IMPL">
* (def-alien-variable "environ" (* c-string))
#<SB-ALIEN-INTERNALS:HEAP-ALIEN-INFO
(FOREIGN-SYMBOL-ADDRESS '"environ") (* C-STRING)>
* (defun posix-environ ()
"Return the Unix environment (\"man environ\") as a list of SIMPLE-STRINGs."
(c-strings->string-list environ))
STYLE-WARNING: redefining POSIX-ENVIRON in DEFUN
POSIX-ENVIRON
* (in-package :cl-user)
#<PACKAGE "COMMON-LISP-USER">
* (posix-environ)
("PWD=/home/dan" "CMUCLLIB=/home/dan" "LISPDIR=/home/dan/src/lisp"
[etc]
Fixes: either
1) define "environ" and POSIX-ENVIRON when the normal toplevel
starts. Icky
2) change the code in run-foreign to do
(def-alien-variable (environ "sbcl_environ") (* c-string))
and add "extern char **sbcl_environ; " and something to initialize it
in the C runtime.
3) something else? is there some kind of late-binding fixup we can
do? my guess is "no", but it's too late at night
Apropos of cCLan, you can see the SBCL bugs (most of them packaging
bugs and/or style issues) that cCLan testers have raised at
http://sourceforge.net/tracker/index.php?group_id=28536&atid=393636
-dan
--
http://ww.telent.net/cliki/ - Link farm for free CL-on-Unix resources

I tried to build my test case to illustrate the problem and
discovered that I was wrong about the problem.
I've just checked in sbcl-0.6.12.46, which has some related
clarifications and comments but few substantive changes.
It's rather impressive how much work DTC did to deal with this and
other issues. I just wish I were better at understanding it..
On Thu, Jul 12, 2001 at 10:00:59AM -0500, William Harold Newman wrote:
> a pointer to the head of the original object, I think the current
> GENCGC hyperconservatism isn't needed.
^^^^^^^^^^^^^^^^^^^^^^^^
[snip]
> current GENCGC hyperconservatism would be needed. Even in languages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[snip]
[etc.]
> > When you made your measurements about the pages pinned at boot (which
> > I deleted, but can't get back since I have a losing mail client), is
> > that memory allocated by the runtime system that we're not allowed to
> > move around, or memory we're not allowed to access in some way? I ask
> > this because I don't recall my SBCL taking up that much memory when it
> > starts.
>
> It's memory that can't be reclaimed in this pass of GC. The GC
> reclaims memory by "moving" (copying; also called "scavenging", more
> or less) all live data out of each page, then freeing the page. When
> there's a live potential conservative pointer to the page, the GENCGC
> deals with it by freezing everything on the page (i.e. not doing the
> "moving"/"scavenging" step, and then not freeing the page).
preserve_pointer() doesn't bogusly pin huge arrays nearly so often
I thought, because the code which'd pin the huge arrays is
downstream from
if (enable_pointer_filter && !possibly_valid_dynamic_space_pointer(addr))
return;
and therefore is usually never executed.
(Here possibly_valid_dynamic_space_pointer() is my new name for
valid_dynamic_space_pointer().) which makes huge arrays much less
problematic than I thought. There may still be a parallel problem for
huge type_CodeHeader objects, noted in the current code as
/* We need to allow raw pointers into Code objects for return
* addresses. This will also pick up pointers to functions in code
* objects. */
if (TypeOf(*start_addr) == type_CodeHeader) {
/* XXX could do some further checks here */
return 1;
}
but (1) the pattern of usage which could tickle that problem
(dynamically allocating and discarding many huge code objects) seems
much less likely and reasonable than allocating and discarding many
huge arrays, and (2) that problem wouldn't explain what's going wrong
with my application. Thus, I'm not all that motivated to worry about
it right now.
The huge pinning statistics reported by GENCGC at startup time seem to
be due almost entirely to valid pointers on the stack, i.e. pointers
which actually refer to the beginning of the array. In that case, the
pinning doesn't seem to do any great harm, just making it a little bit
harder to compact one page. I'd thought that filling the stack with
random numbers would pin any array when any random number pointed into
any interior element, but that's not so, because of the pointer
filtering mentioned above..
--
William Harold Newman <william.newman@...>
The way men live is so far removed from the way they should live
that anyone who abandons what is for what should be pursues his
own downfall rather than his preservation. - Niccolo Machiavelli
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C