[Sbcl-devel] Porting SBCL to Windows

Hi,
I've started trying to port SBCL to Windows.
I'm using the 0.6.19 sources, with Lispworks as the host compiler.
Here's what I've done so far:
- rewrote make.sh, make-config.sh and make-host-1.sh in Lisp,
skipping the part where the "target"-named symlinks are created.
Windows doesn't know what a symlink is anyway.
- setup a few logical-pathname-translations, and replaced the Unix-
specific pathname namestrings with logical-pathname namestrings
(mostly in the files in src/cold, but also in src/misc.lisp, and
of course in stems-and-flags.lisp-expr)
- replaced pathnames of the form "..;target;.." by "..;x86;..".
That should take care of the missing symlinks.
- defined an address space in compiler;x86;parms.lisp
- tried to get Lispworks to load and compile everything, starting
from make-host-1.lisp. With a bit of cheating, Lispworks is willing
to compile up to the file "compiler;info-functions.lisp".
And here are a few things I'm wondering about:
- The type SAP-INT-TYPE is used in cross-sap.lisp and target-sap.lisp.
But it's defined in compiler;x86;vm.lisp, which is too late
(according to the order defined in stems-and-flags.lisp-expr).
Lispworks complains about this, and I think it's right. So I just
copied the type definition from vm.lisp to cross-sap.lisp. But I
wonder how this could have worked using another host compiler.
Or is this a leftover of those famous bootstrapping cross
dependencies I keep reading about?
Would there be a better solution than my stupid copy&paste hack?
- I seem to be having the same kind of problem when compiling
compiler;info-functions.lisp. It needs some definitions from
class.lisp, which seems to need stuff from globaldb.lisp, which
seems to need stuff from info-functions.lisp.
Is that correct? Do you have suggestions for solving this?
- The file Ugliness says that one of the assumptions about the
cross-compilation host Common Lisp is:
SINGLE-FLOAT is distinct from DOUBLE-FLOAT.
I believe that, in Lispworks, SINGLE-FLOAT and DOUBLE-FLOAT are
not distinct. Could you give me an indication where this would
bite me, and what I could do about it?
Thanks,
Arthur Lemmens

Thread view

Hi,
I've started trying to port SBCL to Windows.
I'm using the 0.6.19 sources, with Lispworks as the host compiler.
Here's what I've done so far:
- rewrote make.sh, make-config.sh and make-host-1.sh in Lisp,
skipping the part where the "target"-named symlinks are created.
Windows doesn't know what a symlink is anyway.
- setup a few logical-pathname-translations, and replaced the Unix-
specific pathname namestrings with logical-pathname namestrings
(mostly in the files in src/cold, but also in src/misc.lisp, and
of course in stems-and-flags.lisp-expr)
- replaced pathnames of the form "..;target;.." by "..;x86;..".
That should take care of the missing symlinks.
- defined an address space in compiler;x86;parms.lisp
- tried to get Lispworks to load and compile everything, starting
from make-host-1.lisp. With a bit of cheating, Lispworks is willing
to compile up to the file "compiler;info-functions.lisp".
And here are a few things I'm wondering about:
- The type SAP-INT-TYPE is used in cross-sap.lisp and target-sap.lisp.
But it's defined in compiler;x86;vm.lisp, which is too late
(according to the order defined in stems-and-flags.lisp-expr).
Lispworks complains about this, and I think it's right. So I just
copied the type definition from vm.lisp to cross-sap.lisp. But I
wonder how this could have worked using another host compiler.
Or is this a leftover of those famous bootstrapping cross
dependencies I keep reading about?
Would there be a better solution than my stupid copy&paste hack?
- I seem to be having the same kind of problem when compiling
compiler;info-functions.lisp. It needs some definitions from
class.lisp, which seems to need stuff from globaldb.lisp, which
seems to need stuff from info-functions.lisp.
Is that correct? Do you have suggestions for solving this?
- The file Ugliness says that one of the assumptions about the
cross-compilation host Common Lisp is:
SINGLE-FLOAT is distinct from DOUBLE-FLOAT.
I believe that, in Lispworks, SINGLE-FLOAT and DOUBLE-FLOAT are
not distinct. Could you give me an indication where this would
bite me, and what I could do about it?
Thanks,
Arthur Lemmens

On Wed, Feb 21, 2001 at 10:48:28PM +0100, Arthur Lemmens wrote:
> I've started trying to port SBCL to Windows.
> I'm using the 0.6.19 sources, with Lispworks as the host compiler.
What fun! not only a new target OS, but a few extra idiosyncrasies
from a new cross-compilation host Lisp..
I don't have a Windows machine to test with right now, but no later
than this summer I should have a new laptop, with a shiny new multi-Gb
drive so that it should be pretty painless to leave a few Gb for
Windows when I install OpenBSD or Linux. The only reason I'm delaying
is that that way Moore's Law should give me a few more CPU at the
Computer Go contest at the AGA Congress,
http://www.intelligentgo.org/21cc2001/announcement.html
If you can make a Windows port which is worth merging, I'll stop
procrastinating and get the laptop right away, so that I can work on
merging the port and testing the merged result.
(I hope you'll use, or at least support, a free C compiler for the
src/runtime/ stuff, so that I can build the system without spending a
significant fraction of the cost of my new laptop for Windows-only
development software.)
> Here's what I've done so far:
> - rewrote make.sh, make-config.sh and make-host-1.sh in Lisp,
> skipping the part where the "target"-named symlinks are created.
> Windows doesn't know what a symlink is anyway.
> - setup a few logical-pathname-translations, and replaced the Unix-
> specific pathname namestrings with logical-pathname namestrings
> (mostly in the files in src/cold, but also in src/misc.lisp, and
> of course in stems-and-flags.lisp-expr)
> - replaced pathnames of the form "..;target;.." by "..;x86;..".
> That should take care of the missing symlinks.
> - defined an address space in compiler;x86;parms.lisp
> - tried to get Lispworks to load and compile everything, starting
> from make-host-1.lisp. With a bit of cheating, Lispworks is willing
> to compile up to the file "compiler;info-functions.lisp".
I'm not sure how I feel about turning make.sh into Lisp. It doesn't
seem like a particularly good idea in Unix. I have a little experience
with Windows, but not enough to have a strong opinion whether it's a
good idea there. If the Windows port becomes stable enough to be
merged into the main distribution, I guess I'll have some thinking to
do on this.
The pathnames stuff sounds reasonable. If the Windows port becomes
stable enough to be merged into the main distribution, the symlinks
and physical pathnames should probably go away completely. I can't say
I look forward to that, but in principle it seems to be the right
thing.
> And here are a few things I'm wondering about:
>
> - The type SAP-INT-TYPE is used in cross-sap.lisp and target-sap.lisp.
> But it's defined in compiler;x86;vm.lisp, which is too late
> (according to the order defined in stems-and-flags.lisp-expr).
> Lispworks complains about this, and I think it's right. So I just
> copied the type definition from vm.lisp to cross-sap.lisp. But I
> wonder how this could have worked using another host compiler.
> Or is this a leftover of those famous bootstrapping cross
> dependencies I keep reading about?
> Would there be a better solution than my stupid copy&paste hack?
This works when cross-compiling under SBCL (or CMU CL) because of late
binding, not AFAIK because of any weird cross-dependency. SBCL does
complain about the type not being defined at the time the function is
compiled, issuing warnings at build-the-cross-compiler time. But as
long as the types are defined by the time the functions are executed,
everything works. It's probably inefficient, though, since it probably
compiles into a full call to TYPEP at runtime.
When you say that Lispworks complains, do you mean that it issues
warnings or that it fails?
It should be possible to rearrange things so that the type
declarations appear before cross-sap.lisp and target-sap.lisp. As far
as I can see there's not much point in worrying about cross-sap.lisp,
since a slight efficiency hit at cross-compilation time is no big
deal. But target-sap.lisp stuff ought not be compiled unnecessarily
inefficiently. I'll make a note to look at this. (It's probably easy,
but it goes in the queue sometime behind MNA's big PCL patch and,
probably, the release of 0.6.11, so it might wait a week or more.)
> - I seem to be having the same kind of problem when compiling
> compiler;info-functions.lisp. It needs some definitions from
> class.lisp, which seems to need stuff from globaldb.lisp, which
> seems to need stuff from info-functions.lisp.
> Is that correct? Do you have suggestions for solving this?
What kind of definitions? Type definitions or other definitions?
And what kind of problem? Warnings or failures?
I spent a long time, a long time ago, untangling the order of things
enough to let the system build itself without ever trying to use
something which wasn't defined yet. I left many places where it
compiles a forward reference to something which isn't defined yet, and
at least some of those should be untangled, since such forward
references can be much less efficient than the optimized/inlined
version. But as far as I know everything should work, even if not
always as efficiently as possible.
> - The file Ugliness says that one of the assumptions about the
> cross-compilation host Common Lisp is:
> SINGLE-FLOAT is distinct from DOUBLE-FLOAT.
> I believe that, in Lispworks, SINGLE-FLOAT and DOUBLE-FLOAT are
> not distinct. Could you give me an indication where this would
> bite me, and what I could do about it?
It might bite you in constant folding in the cross-compiler. If so, it
might be fixable by suppressing constant folding at cross-compilation
time.
It will probably be a problem when dealing with DEFCONSTANTs of
floating point values in the ordinary style like this,
(defconstant pi 3.14159265358979323846264338327950288419716939937511L0)
but I notice that most of them seem to be in another style,
(defconstant least-positive-double-float (double-from-bits 0 0 1))
and that style should be OK as long as DOUBLE-FROM-BITS isn't
constant-folded. So if necessary, you can probably hack things
like DEFCONSTANT PI into this form as well and make stuff work.
A similar problem would arise if there are any other places where the
cross-compiler wants to dump DOUBLE-FLOAT values. I can't think of any
offhand, but it might happen that the cross-compiler generates a
DOUBLE-FLOAT value and dumps it, expecting the distinction between
DOUBLE-FLOAT and SINGLE-FLOAT to be preserved in the cross-compiler
and appear in the target.
It might in principle be a problem in optimization of DOUBLE-FLOAT
expressions at cross-compile time, since the compiler tries to do
clever things like trying to infer the type of the results of some
functions from the types of their arguments. However, in practice I
doubt that's a problem, since I can't think of any place in the build
process where the cross-compiler processes expressions subject to that
kind of optimization, and the cross-compiler doesn't need to be
correct for all possible Common Lisp input, only for the subset of
Common Lisp which is actually used to define SBCL itself.
Those are all the gotchas that I can think of, but without testing it,
it's hard to be sure that there aren't any more. If you're worried
about it, you could probably test it (without mixing up too many
Windows interface issues) by using Lispworks as a cross-compilation
host for a Unix version of SBCL, and make sure that that works.
--
William Harold Newman <william.newman@...>
software consultant
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

William Harold Newman <william.newman@...> writes:
> I'm not sure how I feel about turning make.sh into Lisp. It doesn't
> seem like a particularly good idea in Unix. I have a little experience
> with Windows, but not enough to have a strong opinion whether it's a
> good idea there. If the Windows port becomes stable enough to be
Windows batch files have all the expressive power of daytime radio, so
I'd guess it's a choice between that or install some kind of
third-party shell (perl, bash, whatever)
What about turning it into a Makefile? With five or so phony targets
(clean, host1, target1, host2, target2) it wouldn't exactly be a
normal use of make, but it would probably do the job. And some kind
of make utility (as things stand currently, GNU Make and none other)
will be needed to build the runtime anyway.
-dan
--
http://ww.telent.net/cliki/ - Link farm for free CL-on-Unix resources

Daniel Barlow <dan@...> writes:
>
> Windows batch files have all the expressive power of daytime radio, so
> I'd guess it's a choice between that or install some kind of
> third-party shell (perl, bash, whatever)
What about cygwin? The install process is so easy it's scary, and it
does a good job of translating between Unix / Windows idiosyncrasies
wrt directory structure etc. From what I see, the build process could
just stay the same then.

William Harold Newman <william.newman@...> writes:
> On Wed, Feb 21, 2001 at 10:48:28PM +0100, Arthur Lemmens wrote:
> > I've started trying to port SBCL to Windows.
> > I'm using the 0.6.19 sources, with Lispworks as the host compiler.
>
> (I hope you'll use, or at least support, a free C compiler for the
> src/runtime/ stuff, so that I can build the system without spending a
> significant fraction of the cost of my new laptop for Windows-only
> development software.)
The cygwin port of gcc compiles native windows apps with -mno-cygwin.
bash, gnu make etc. are also available. With the cygwin tools, the
build process would be the same as under the unices.
I started thinking about how to port the C runtime to windows about a
month ago, and here's what I came up with:
Memory handling: os_validate() and friends can be written with
VirtualAllocEx() and CreateFileMapping() / CreateViewOfFileEx().
Exception handling: signals can be faked with
SetUnhandledExceptionFilter(). The EXCEPTION_RECORD and CONTEXT
structures seem to have all the run-time information that is needed.
I have sketched some code, but nothing compilable yet. In march, I
will have some more time to work on it. Could this C runtime, as soon
as it's written, read a SBCL core file generated on Linux, assuming
the correct lisp.h was linked in?
Rudi

On Thu, Feb 22, 2001 at 01:23:19PM +0100, Rudolf Schlatte wrote:
> I have sketched some code, but nothing compilable yet. In march, I
> will have some more time to work on it. Could this C runtime, as soon
> as it's written, read a SBCL core file generated on Linux, assuming
> the correct lisp.h was linked in?
It depends on what you mean. If you mean could you take the
core file you're running now and use it on Windows, the answer
is almost certainly "no":
1. SBCL needs to know the sbcl.nm link map at GENESIS time, because
it implements calls to runtime functions by jumping to the
address read from the nm file at this time. In order to make
the core work, you'd need to arrange for every C address which
the core references directly to be in the exact same location
in Windows as it was at GENESIS time on Linux. This would be
much, much more trouble than it's worth.
2. SBCL also needs to know the overall memory map at build time,
because things like the location of NIL and the boundaries of the
GCable spaces get hardwired into the code. It is unlikely
that the memory map on Windows can be made identical to the
memory map on Linux.
3. Various things, e.g. (SIN X), get inlined into native calls into
the C library. You'd need the exact same functions, with the exact
same calling convention.
4. Several OS-interface files, especially unix.lisp, work by
hand-copying constants and structure layouts from system header
files. It is unlikely that these are the same on Windows.
If you want to use Linux to generate a core file specifically for
Windows, it's less hopeless, though still unlikely. Problem #1
disappears if you take the nm file generated by Windows for the new
Windows runtime, copy it to Linux, use it as input for GENESIS, and
copy the resulting core file back to Windows. Problem #2 disappears if
you tell GENESIS about the Windows memory map. With problems #1 and #2
gone, there's some small chance that the core file might work, at
least a little bit. But I expect you'd still need to do some work, and
probably a significant amount of work, to address problems #3 and #4.
--
William Harold Newman <william.newman@...>
software consultant
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C

William Harold Newman <william.newman@...> writes:
[generating core files on linux for windows]
>
> 3. Various things, e.g. (SIN X), get inlined into native calls into
> the C library. You'd need the exact same functions, with the exact
> same calling convention.
> 4. Several OS-interface files, especially unix.lisp, work by
> hand-copying constants and structure layouts from system header
> files. It is unlikely that these are the same on Windows.
For very small values of "unlikely". The windows Posix layer isn't.
> If you want to use Linux to generate a core file specifically for
> Windows, it's less hopeless, though still unlikely. Problem #1
> disappears if you take the nm file generated by Windows for the new
> Windows runtime, copy it to Linux, use it as input for GENESIS, and
> copy the resulting core file back to Windows. Problem #2 disappears if
> you tell GENESIS about the Windows memory map. With problems #1 and #2
> gone, there's some small chance that the core file might work, at
> least a little bit. But I expect you'd still need to do some work, and
> probably a significant amount of work, to address problems #3 and #4.
Yup. The idea was to basically do all make-host-* steps on Linux and
all make-target-* steps on Windows, hand-transferring the
memory-map-related thingies between the systems. I figured
(optimistically) that this would bring the system to a point where one
could watch it load the core and die in some call-into-c spot.
Once that point is reached, I would start thinking about some generic
layer on the lisp side that would use sb-unix.lisp or the
to-be-written sb-windows.lisp and abstract them both away. But that's
not even in the planning stage.
Rudi

Rudolf Schlatte <rschlatt@...> writes:
> > 4. Several OS-interface files, especially unix.lisp, work by
> > hand-copying constants and structure layouts from system header
> > files. It is unlikely that these are the same on Windows.
>
> For very small values of "unlikely". The windows Posix layer isn't.
Ho ho. Have you looked at the definition of struct stat in there
lately?
(I don't wish to be discouraging - it's not insuperable, just tedious)
-dan
--
http://ww.telent.net/cliki/ - Link farm for free CL-on-Unix resources

William Harold Newman wrote:
> that way Moore's Law should give me a few more CPU at the
> Computer Go contest at the AGA Congress,
> http://www.intelligentgo.org/21cc2001/announcement.html
You've written a Go program? Wow! How strong is it?
Using Common Lisp must give you a big advantage. Last time I looked,
most guys were using C++, assembler and other atrocities for their
Go programs.
> I'm not sure how I feel about turning make.sh into Lisp. It doesn't
> seem like a particularly good idea in Unix. I have a little experience
> with Windows, but not enough to have a strong opinion whether it's a
> good idea there. If the Windows port becomes stable enough to be
> merged into the main distribution, I guess I'll have some thinking to
> do on this.
It's no big deal. There are only a few lines of code in those files
anyway.
> The pathnames stuff sounds reasonable. If the Windows port becomes
> stable enough to be merged into the main distribution, the symlinks
> and physical pathnames should probably go away completely. I can't say
> I look forward to that, but in principle it seems to be the right
> thing.
Using logical instead of physical pathnames doesn't have to have
many consequences. You can still use string concatenation to
construct pathnames. All I did was: set up a few logical pathname
translations, add a logical host in a few places, and replace slashes
by semicolons.
Arthur

On Fri, Feb 23, 2001 at 11:15:54PM +0100, Arthur Lemmens wrote:
> William Harold Newman wrote:
>
> > that way Moore's Law should give me a few more CPU at the
> > Computer Go contest at the AGA Congress,
> > http://www.intelligentgo.org/21cc2001/announcement.html
>
> You've written a Go program? Wow! How strong is it?
> Using Common Lisp must give you a big advantage. Last time I looked,
> most guys were using C++, assembler and other atrocities for their
> Go programs.
My program is still very weak. The strongest version so far had a lot
of clever code to try to understand influence and territory, but
essentially no tactics reading. It got crushed at the AGA tournament
in Denver two and a half years ago, and I set aside Go programming for
a while to work on getting CMU CL's bootstrapping unscrewed, an effort
which consumed almost all of my non-contracting programming efforts
until sbcl-0.6.0 or so. The new version has what I think is a very
promising tactics engine, but I've only begun feeding knowledge into
it, so at the moment it's still only about as strong as the Denver
version.
As to languages, we can hope it will turn out that way, but I can't be
sure. In a world full of people trying to teach computers to play Go,
including various abstraction-philic computer scientists and
mathematicians writing not only programs but also papers and theses
and books about the problem, the field is still dominated by low-level
languages, including for several years recently a single dominant
program (Handtalk) written in 8086 assembly language. (So perhaps I
should pin my hopes on my chemistry background instead, since
Handtalk's author was a retired chemist.:-)
(And actually I rather like C and C++. When microefficiency is
important, it can be substantially less ugly just to write the program
in C than to try to squeeze the efficiency out of a more abstract and
dynamic language. And if the code has complex behavior, C++ can be
substantially less ugly than trying to improvise objects and
exceptions and whatnot in C. I just don't like to use C or C++ for
code that's going to be redesigned a lot, which is to say at least 50%
of the code I've seen, not only 50+% of the throwaway and prototype
code, but also 50+% of the production code. And of course, even within
the remaining fraction, microefficiency is often not all that
important.)
--
William Harold Newman <william.newman@...>
software consultant
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C