0. Availability

The most recent version of this document is available at
http://squeakvm.org/squeak64/.
(If the link is broken, it means this document is no longer
relevant.)

VM source code, 32- and 64-bit images, and pre-compiled 32- and
64-bit VMs for a variety of 32- and 64-bit platforms are available
at
http://squeakvm.org/squeak64/dist3.
(If the link is broken, or the directory empty, it probably means
64-bit support has been pulled into the binaries that are
available through the usual Squeak distribution channels.)

ChangeSets is a directory containing stuff you might want to
file into your image, if you aren't going to be using one of the above
images. Within this directory:

VMM38b4-64bit-image1-ikp.1.cs and
VMM38b4-64bit-image2-ikp.1.cs contain 64-bit compatibility
for the Squeak class library. They must be filed-in in the order
shown.

System-Tracing.2.cs contains a SystemTracer capable of writing a 64-bit image.

VMMaker-tpr.14.mcz contains the latest (as of this
writing) VMMaker from Tim, to which all the 64-bit changes are
relative. File this in first, before either of the following.

VMM38b4-64bit-vm1-ikp.1.cs contains 64-bit compatiblity
for the CCodeGenerator, ObjectMemory, core Interpreter, and the three
essential plugins: BitBlt, BalloonPlugin, and FilePlugin. File this
in before the next changeset.

VMM38b4-64bit-vm2-ikp.2.cs contains 64-bit compatiblity
for the SocketPlugin and the SmartSyntaxInterpreterPlugin (and
corresponding code generator). File this in last.

With all of the above filed in you should have images identical to
those in the archives above, capable of generating both 32- and 64-bit
VM sources and of being traced from a 32-bit image to a 64-bit image
that will load and run.

Pretty much everything else is as described in the standard documentation.

1. Introduction

This is a "port" of Squeak to 64 bits, where "porting" involves
two related (but quite separable) tasks:

Image format changes to support 8-byte pointers, with
corresponding modifications
to ObjectMemory, Interpreter
and BitBltSimulation.

CCodeGenerator and runtime support changes,
to make everything 64-bit clean.

(In the remainder of this document, the words "porting" and
"ported" are meant to include both of these areas and imply making
the code compatible with both 32- and 64-bit images. Note that in
some cases, image-side work in the former area implies runtime
support work in the latter area; this is especially true for
plugins.)

Generally speaking, Dan Ingalls worked on the former while Ian
Piumarta worked on the latter.

Keeping the two areas properly separated leads to a significant
benefit: the Squeak VM can now be compiled four ways, to run all
four combinations of 32- or 64-bit images on 32- or 64-bit
hardware. Most importantly, this allows volunteers helping with
the plugin "porting" effort to test both 32- and 64-bit VMs
regardless of the kinds of hardware to which they have access.
(While porting BitBlt and B2D it was actually
quite satisfying to see precisely the same rendering glitches on
both 64- and 32-bit hardware running a 64-bit VM. :-)

1.1. Status

There is still much work (and cleaning up) to be done. Here is a
brief list of what we have so far:

A new image format, minimally different from the 32-bit format.
The SystemTracer has been extended
to clone the running image in this 64-bit format.

The core VM (ObjectMemory, Interpreter
and BitBltSimulation) have been modified to execute both
32- and 64-bit images, depending on the setting of a class
variable
in ObjectMemory. FilePlugin, BitBltPlugin
and much of B2DPlugin have also been ported. All of the
cross-platform common code on which these parts of the VM depend
has been made 64-bit clean. In other words, the minimum
functionality required to load an image, execute bytecodes, access
changes/sources files, and display the UI has been
ported.

The corresponding parts of the Unix support code have been ported:
core VM support, platform-dependent code for the above plugins,
and the display drivers for X11 and Quartz. The only
architecture/compiler requirement is that both 32- and 64-bit
integer types be available. (This means that 32-bit machines
must implement the "LL64" model where the long long
int type is 64 bits wide, and that 64-bit machines must
implement the "LP64" model where the int type remains
32 bits wide. See
http://www.opengroup.org/public/tech/aspen/lp64_wp.htm
for an explanation of these acronyms.) Note that all plugins
are currently compiled "internally". (External plugins should
work, but have not been thoroughly tested.)

The above has been tested on Mac OS X (PowerPC, G4 and G5) and
GNU/Linux (Itanium and Pentium).

A significant side effect of this work is that
the InterpreterSimulator has been fixed and extended to
simulate both 32- and 64-bit images.

1.2. Remaining work

The single most important thing is: pull the 64-bit changes into
the mainstream Squeak release. Before that can happen for a
"stable" release, at least the following must be addressed:

Most of the plugins have not been ported. In
particular, LargeIntegerPlugin and SocketPlugin
are not yet available. One of us (Ian) has volunteered to port
the SocketPlugin (along with asynchronous files and
pseudo ttys). Hopefully the Squeak community will volunteer to
port the remaining plugins.

sqMemoryAccess.h uses some C99 features that have been
available in gcc for decades but which are possibly
unavailable in other compilers. In particular, "static
inline" functions therein should be rewritten as macros.

Desirable, but not essential, are:

Some changes have still not been made in the image and in the
support code, mostly relating to 64-bit Bitmaps and access
to 64-bit integers within them.

The mechanism used to communicate the width of an oop between the
generated source code and the platform-dependent code is ugly and
redundant. Identical source code should be generated for both 32-
and 64-bit VMs; the width of an oop supported by the final
compiled VM should be a (C) compilation time option, not a
(pre-)VMMaker source code generation option.

Some mechanism is needed to allow older VM sources
(using int for everything) to compile within the new
framework.

Some mechanism is needed to allow "32-on-64" mode compilation when
running "32-on-32". This would cure the famous problem of the
heap being allocated above the 2 GB boundary on 32-bit machines
(such that oops become negative).

2. 64 bits the easy way

You can download a 64-bit image and a precompiled binary for Mac
OS X (PowerPC) or Linux (Itanium or Pentium) from the site
mentioned in Section 0.

Squeak64-image.tar.gz contains a 64-bit image and
its changes file.

Squeak64-3.7x-5.your-architecture.tar.gz
contains the VM.

3. 64 bits the hard way

The "honest" way to obtain a 64-bit image and VM is to build them
yourself. Again, everything you need is available from the site
mentioned in Section 0.

3.1. Generating a 64-bit image

You will need a 3.7 image (the following has been tested and is
known to work with the Squeak3.7-5989-full image) into
which you have loaded the System-Tracing.1.cs changeset.

Evaluate the following in a workspace to clone the running image
into a 64-bit equivalent:

Because of the "half-initialised" state of a cloned image, it is a
good idea to start it up and save it immediately after cloning (in
particular, before moving it out of the current working
directory).

3.2. Building a 64-bit VM

You will need the modified Unix platform
sources Squeak-3.7x-5.src.tar.gz and
the All64BitChanges.1.cs change set.

The All64BitChanges change set should contain
everything needed to add 64-bit compatibility to the generated
sources. (Again this has been tested, and is known to work, with
the 3.7-5989 full image.)

We did not modify VMMaker's user interface. However, the
generated sources are currently different depending on the word
size (4 or 8 bytes) desired. To select between the two possible
sizes, evaluate one of the following expressions in a workspace:

ObjectMemory initBytesPerWord: 4; initialize.

sets things up to generate a 32-bit VM, while

ObjectMemory initBytesPerWord: 8; initialize.

sets things up to generate a 64-bit VM. (This only needs to be
done once, when changing from one word size to the other.)
Pressing the "clear out" and then "generate all" buttons in
VMMaker will generate the sources as usual.

The VM can now be built in the traditional manner: create a build
directory, descend into it, and then run configure
followed by make.

3.3. Step-by-step guide

If the above is a little too terse, you can follow these steps
to create your own 64-bit image and VM "from scratch":

Download the Unix sources Squeak-3.7x-5.src.tar.gz
from the above site.

Unpack the sources and descend into the directory created.

Download the two change sets (System-Tracing.1.cs
and All64BitChanges.1.cs) from the above site to the
current directory.

Download the Squeak3.7-5989-full image and changes
files from one of the usual Squeak sites into the current
directory.

Make a link to (or a copy of) SqueakV3.sources in the
current directory.

Start the image with your usual VM.

File-in the two change sets. (I usually
file-in System-Tracing followed
by All64BitChanges, but the order should not
matter.)

If it works, read the next section and volunteer to port a plugin or two!

To build a 32-bit VM from the same sources, follow a similar procedure but evaluate

ObjectMemory initBytesPerWord: 4; initialize.

before pressing "clear out" and "generate all" in VMMaker.

3.4. Caveats

Building a 64-bit VM on the Pentium confuses gcc-3. Use 2.95
instead. (This problem does not seem to affect any other
architecture.)

4. Hacking the VM

The following guidelines should help keep the VM and plugin code
working for both 32- and 64-bit VMs running on 32- and 64-bit
hardware:

Don't assume any relationship between the widths of
oops, integers, and pointers. There is no guarantee that any
two of these will have the same width.

Never use explicit casts to convert between integers or
oops and pointers.

Only store pointers in pointer-typed variables. (Similarly
for pointer arguments and return types; be completely honest
with function prototypes.) Using "int" as the
"anonymous type" in any situation will have disastrous
consequences.

Ensure that integer (and oop) types are of the
appropriate width (see below).

Two new types sqInt and usqInt are provided to
help with this:

type

width

[unsigned] char

8 bits

[unsigned] short

16 bits

[unsigned] int

32 bits

usqLong, sqLong

64 bits

usqInt, sqInt

same as oop

sqInt and usqInt are defined according to the
word size selected when the core VM sources are generated (as
described above) and the widths of the available primitive types
(as determined by the configure script). sqLong
and usqLong are currently unused (pending the completion
of 64-bit Bitmaps).
Note that the CCodeGenerator now generates code in which
all implicitly-typed quantities (i.e., oops) are declared
as sqInt.

The traditional memory access macros have been extended, in the
obvious manner with respect to the above types, as follows:

macro

use

byteAt: anOop [put: aValue]

read/write an 8-bit integer

shortAt: anOop [put: aValue]

read/write a 16-bit integer

intAt: anOop [put: aValue]

read/write a 32-bit integer

oopAt: anOop [put: aValue]

read/write an object pointer

longAt:[put:] is currently synonymous
with intAt:[put:] (for backward compatibility
with the vast majority of memory accesses in unconverted code) and
should be eliminated during any conversion effort. (At some point
in the near future, longAt: will refer to a 64-bit
integer, corresponding to the sqLong type.)

The above all access memory based on an object pointer (which is
an integer quantity, not a pointer). To access memory
given a pointer to the value to be read/written, use one of the
corresponding macros (of which only two are shown here, but they
are available for all supported value types):

macro

use

byteAtPointer: aPointer [put: aValue]

read/write an 8-bit integer

...

oopAtPointer: aPointer [put: aValue]

read/write an object pointer

Conversion between an oop and a "real" pointer into memory must be
performed using the two macros provided:

macro

use

pointerForOop: anOop

answer the address referenced by anOop

oopForPointer: somePointer

answer an oop corresponding to somePointer into memory

(The oop-based accessors are in fact written in terms of the
pointer-based accessors, using the pointerForOop: "cast"
macro to convert from their oop argument to a memory address.)

The traditional "cast" macro:

cCoerce: anyValue to: typeString

should be avoided unless absolutely necessary (and never
used to coerce between integer and pointer values).

4.1. Cross-platform changes

A new file, "interp.h", is generated by VMMaker. This
file defines the cpp macro SQ_VI_BYTES_PER_WORD
as 4 or 8 according to the "bytesPerWord"
setting in effect at the time the sources were generated.

A new file has been added:
"platforms/Cross/vm/sqMemoryAccess.h". This file defines
the types and macros described above. Note that the traditional
memory access macros (byteAt:, longAt:, etc.)
are no longer defined explicitly at the top of every generated
source file. Any source that might need them must
include sqMemoryAccess.h instead.

4.2. Who to call for help

We are happy to give guidance (and assistance in using the
features described above) to anyone converting (or writing from
scratch) VM or plugin code. Our e-mail addresses are:
firstName at squeakland dot org