Archived stand-alone executables, with most `contrib' modules preloaded. They may be useful when normal software installation is not allowed for your account, but running unverified executables is allowed.

ATTENTION: those executables include some useful REPL extensions, so there may be a reason to prefer them; however, source code implementing the extensions is not published yet. The latter may be a reason to avoid the executables, or to contact me and ask for source access.

Introduction

Dmitry Kalyanov has done the great work of adding threading support to
Steel Bank Common Lisp (SBCL) for Windows, and continues to improve on
it. While this code is not added to official SBCL code base yet, I've
started to use it in a real project as soon as it became stable
enough.

SBCL for Windows is still experimental, and there are many places in
need for platform-specific fixes. There are also some changes required
specifically for threaded build: e.g. on other platforms where SBCL
supports threading, blocked I/O operation will be interrupted by
sb-thread:interrupt-thread, but current official implementation of
win32 I/O (combined with safepoint-based interrupts used by Dmitry)
doesn't allow it.

Therefore I created my own fork of Dmitry's GIT repository and started
to work on Windows-specific issues, beginning with the imminent ones.

Dmitry and I merge our changes periodically, to lower the probability
of long, boring conflict-resolution work in the future. His
experimental tib-3 branch includes my changes, except a few ones done
after the last merge.

This text is purported to provide a description of the problems I've
tried to solve and of the solutions chosen. Sometimes I describe the
peculiarities of Windows in depth; it could be boring for experienced
Windows programmer, but I hope that it will be useful for SBCL
maintainers working on normal platforms.

Windows is Special

This section describes the aspects where Windows platforms are
different from Unix-like ones (the latter are traditional, widely-used
targets for SBCL and its ancestor, CMUCL). SBCL internals and my
changes are mentioned only in passing here.

What is Win32 and Is There Win64

There is a longstanding tradition of using "32" as an antonym for "16"
in the Windows world. SBCL will probably be ported to 64-bit Windows
systems in the future (if those systems survive until that time). Due
to a peculiar meaning of "32" described above, unofficial terms like
"Win32 API" are applied to those systems as well. Names of the system
DLLs implementing the API still end with "32", too.

Therefore, most of the notes will be applicable to 64-bit Windows
(only a few problems, like multiple calling conventions, are specific
to 32-bit systems). And I would advise SBCL developers to retain
:win32 in *FEATURES* for 64-bit port, following the tradition of
Windows world in general.

Win32 History in a Nano-Nutshell

Initially there was two implementations of what is called "win32 API",
providing the same interface with entirely different internals:

NT implementation, on top of 32-bit, multitasking, unicode-aware
Windows'NT;

Win32s subsystem on WfW 3.11 and its glorified heirs on Windows
95/98/ME.

Modern Windows systems for PC (Windows'2000,XP,2003,Vista,Windows7...)
are based on the Windows NT variant; the 95/98/ME generation's install
base is already unnoticeable, and they're neither sold nor supported
by the vendor any more.

Publicly available documentation on MSDN now reflects this fact by
covering compatibility of the API functions only within NT family, for
Windows'2000 and above. Everything before Win2k is antediluvian and
isn't worth mentioning. Targetting pre-Win2k systems in our times is
problematic in more than one way, and the usefullness of this endeavor
is decreasing every day, if visible at all. Therefore it would be wise
to limit our compatibility and porting efforts to Win2k and above.
The next big bunch of API improvements arrived with Windows Vista, but
older systems are still widely used, so I think it's too early to rely
on these features unconditionally in a project like SBCL.

Since Windows XP, each major change in the OS codebase ends up in two
flavors of Windows, marketed as two products with different names:
"server" and "client" OS variants. From the programmer's point of
view, both variants in each pair are almost the same:

Windows XP and Windows Server 2003

Windows Vista and Windows Server 2008

Windows 7 and Windows Server 2008 R2

MSDN documentation includes both client and server versions in its
Compatibility sections for function descriptions.

Plain C Programmer's View: Ogres Have Layers

When we use printf or fputs on most platforms, we are calling a
function residing in the C runtime library (CRT). On Windows, there is
a canonical implementation of C runtime library, MSVCRT.DLL
(i.e. Microsoft Visual C runtime). It was initially included into
redistributable package coming with MSVC compilers; now it's
officially a part of the OS. Newer versions of MSVC use runtime
libraries with other names (for dynamically-linked runtime, DLL has a
versioned name like MSVCR71.DLL).

SBCL for Windows is compiled with MinGW. The latter is a port of GCC
that is capable of building "native" applications, linked against
MSVCRT.DLL by default. Since MSVCRT.DLL is pre-installed with Win2k or
above, such applications don't require any redistributable runtime
package to be installed: they work out of the box on any modern
Windows system.

MSVCRT itself is implemented on top of "Win32 API" functions, provided
by KERNEL32.DLL, USER32.DLL and other system libraries. Those
libraries are supposed to be used by applications directly, when
OS-specific features beyond basic file I/O are needed. Interfaces
provided by them are stable and well-documented (well, comparatively);
backward compatibility was mostly retained since pre-unicode, pre-NT times
(WfW 3.11 + Win32s, Windows'95/98/ME) and early Windows'NT (3.x and above).

As of the NT family, system DLLs interact with OS kernel and drivers
with so called "native API", which is a thin layer on real kernel
system calls. Native API is available for applications as a functions
exported from NTDLL.DLL. Some parts of it are documented as part of
Windows DDK, other parts are undocumented (but some undocumented parts
are still widely used). There are things that are impossible to do
without native API, and some of them are potentially useful for
SBCL. So far I was able to go without these functions, but I may
decide to use them in the future. Given the pragmatic nature of
Microsoft's policy regarding backward compatibility, calling a
widely-used undocumented function is almost as safe as calling an
officially documented one.

There is also a socket implementation (winsock2), WS2_32.DLL. Concrete
protocols and address families are hooked into it in a plugin-like
fashion. A set of BSD-like socket-related functions is provided (they
work almost like the real thing, errr, BSD-style API on Unices), as
well as other Winsock-specific or Windows-specific functions. Winsock2
was initially intended to be portable (though I don't know of any
non-Windows implementation); some entities provided and used by
Winsock2 are documented (or known) to be the aliases of corresponding
Windows entities for Windows implementation:

WSACreateEvent and CreateEvent work with the same kind of events
(officially);

WSAGetLastError and GetLastError work with the same variable
containing an error code (known);

Each socket handle may be used as a Windows file handle for NT
family (officially).

Though full socket I/O support doesn't belong to core SBCL runtime,
there are things that require Winsock2 calls when an open I/O channel
happens to be a socket. As SBCL process may inherit a socket handle as
its standard input or output (though this approach is not used on
Windows as frequently as on Unices), core SBCL runtime still has to use
Winsock2 and link against it.

Streams, Descriptors, Handles and Objects

MSVCRT itself, with regard to I/O operations, contains two layers at once:

Each stdio stream (FILE*) references a lowio descriptor. Each
descriptor, in turn, is an index into a table inside CRT, containing
Windows file handle as well as other useful information. If we go on
with this story (going into the native API), each file handle is
either a kernel-level handle, or a console handle (console handle
operations are intercepted by KERNEL32.DLL and redirected to CSRSS.EXE
using client-server remote procedure calls). Some kernel-level file
handles are also Winsock socket handles.

Any handle of a kind discussed so far may be duplicated with
DuplicateHandle() (though there are some problems for socket handles
that will be discussed below). When a handle is duplicated, it still
designates the same kernel object; the only private thing that it has
is a possibility of inheritance.

As the lowio functions are almost a drop-in replacement for their Unix
counterparts, SBCL developers decided to use it as an emulation
layer. It is a non-trivial design decision (many other language
runtimes preferred raw file handles on Windows as an equivalent for
Unix descriptors). This decision has its merits, especially for Common
Lisp implementation: for MSVCRT lowio, 2047 is a maximum possible
value for a descriptor, so all valid descriptors are fixnums.

There is a way to wrap an existing file handle into a lowio descriptor
(_open_osfhandle) and to get it back (_get_osfhandle). However, after
a file handle is associated with a descriptor, there is no way to
destroy or alter this association without closing the descriptor (the
handle is closed with the descriptor unconditionally, too).
Closed descriptors are associated with INVALID_HANDLE_VALUE
((HANDLE)-1).

Standard Input, Output and Error

As of Win32 API layer, standard channel handles are set and retrieved
using SetStdHandle and GetStdHandle function, respectively. When
MSVCRT.DLL is loaded, it associates those handles with descriptors 0,
1 and 2, and creates FILE* stdin, stdout and stderr ANSI streams.

If SetStdHandle is used later, associations for descriptors 0, 1 and 2
remain the same.

Values of standard channel handles are specified when the process is
created (CreateProcess in Win32 API), or simply inherited from the
parent process when appropriate flag is given.

On Unix there is a tradition, not mandated but widely respected, to
make normal processes inherit descriptors 0, 1 and 2 (pointing to
/dev/null if there is no other sensible input or output). On Windows,
the same stands for console applications (including the fact that it's
not mandated or enforced).

Locking and Serialization

When threads are used, the consequences of doing things on the same
channel concurrently suddenly become important. Newer MSVC runtimes
introduce some locking on stdio level, along with non-locking
counterparts. As of the lowio level, there is no locking for the
duration of an operation itself, but internal MSVCRT descriptor table
is protected enough for concurrent access to be safe: no corruption
will occur on attempt to _dup2 into the same target descriptor
concurrently; only one of concurrent _close() calls on the same lowio
descriptor will succeed, etc.

Calls to lowio _read() and _write() are translated into Win32 API
synchronous I/O calls ReadFile() and WriteFile() without any
additional locking at the CRT side for the duration of I/O itself.
Therefore, further discussion of locking and serialization will be
concerned with the behavior of Win32 API routines, not their lowio
wrappers.

Synchronous I/O operations that are used by lowio code are serialized
by the OS kernel itself (or by KERNEL32.DLL for console handles).
There are some useful aspects in it for C application developers, but
there are also some problems that made me avoid _read() and _write()
completely at the end.

The main problem is that kernel object locking is not
direction-specific: outstanding read operation causes concurrent write
operation to block. The important aspect is that it's the kernel
object that is locked, not a handle; consequently,

duplicating the handle will never solve locking problems,

if your process isn't doing anything with a handle, it may still be
locked by another process.

It's not uncommon for Common Lisp projects to use a separate "writer"
and "reader" threads for the same bidirectional stream (SLIME works
this way with socket streams in multi-threaded mode; my own in-house
project works this way with serial ports). We have to support it,
so this kind of locking granularity is too large for us.

Another problem with synchronous I/O is its synchronous nature: when a
thread is blocked, it can't be interrupted. This problem has a couple
of interesting solutions while keeping the operation synchronous: one
solution uses new functions introduced in Vista, another one uses
native API. I haven't implemented any of them for SBCL yet, but it
might make sense: with some kinds of handles we have no other option
but the synchronous I/O, so some operations still cannot be
interrupted.

OVERLAPPED (Asynchronous) I/O Facility

Win32 API provides a special I/O mode, called OVERLAPPED I/O:
application shall issue a request to start an operation; when the
operation completes, OS notifies the application. Any outstanding
operation may be cancelled by the thread that initiated it; it's also
cancelled automatically if the thread exits. Overlapped call doesn't
block even if there is another outstanding operation on the same
kernel object in another thread. However, an operation itself is done
atomically: when two blocks of data are written simultaneously, one of
them won't end up in the middle of the other.

It seems that it's exactly what we need (and what I'm using now when
it's possible). There are some problems surrounding this solution,
however, arising both from the Win32 API side and the CRT side:

OVERLAPPED mode is enabled or disabled for kernel objects at
creation time, and it can't be toggled afterwards. If our process
inherits a handle, it's either OVERLAPPED or not (usually not —
and there's a reason why). Our process has to live with it.

For on-disk files, the file offset where the operation should start
is specified in completely different way for OVERLAPPED calls.

Console handles are never in OVERLAPPED mode (remember, they are
not "real" handles from the native API point of view).

Anonymous pipe handles, though they are real kernel handles, aren't
OVERLAPPED as well.

Event Objects

Overlapped I/O completion can be signalled in two ways: with a queued
callback (APC), or with an event object. However, we have to work with
file handles of both OVERLAPPED and non-OVERLAPPED kinds, and there is
no simple way to request this property for a handle (native API has to
be used for it). ReadFile() and WriteFile() provides such an
"agnostic" solution: they fall back to synchronous mode if an
OVERLAPPED structure is given but the handle is not OVERLAPPED. Event
object, however, is the only way that those functions may use to
notify the completion of an asynchronous operation.

Event objects are kernel objects, not unlike a sort of dumbed-down
boolean semaphores. As far as I know, they are unique for Win32 API
and have no equivalents in other modern systems. They are also
notorious for looking as a right thing to use when they aren't: race
conditions, missing signals and fairness problems are to be expected
around them. I've tried to be careful this time, but I'm also
notorious for misusing event objects, so watch out.

Waitable Timers

Implementation of (SLEEP) that can be interrupted and continued
requires something other than Sleep or SleepEx call (the latter may be
interrupted with APC if it's alertable, but can't be
resumed). Waitable timer objects provided by Win2k and above are the
thing that I decided to use (for threaded builds only, by now, but
they would work on non-threaded builds as well, so maybe it's better
to unify the code: pre-Win2k portability is already unattainable).

Kinds of I/O Handles

Some operations that are unified for Unix descriptors (like checking
if a future read() operation would block) don't have lowio
equivalents; as of Win32 API equivalents, they are sometimes available
for a particular kind of a file-like object, or for some kinds with
different things to do for each one.

SBCL code sometimes check for the following kinds of file-like objects
to handle them specially:

Error reporting

MSVCRT provides a traditional errno "variable" (it's actually a macro
expanding into dereferencing the result of a function call, allowing
this thing to be thread-local). Symbolic constants for error codes are
defined in errno.h, and a corresponding textual message may be
retrieved with strerror().

Win32 API returns its error codes in entirely different place,
available with GetLastError() and SetLastError() (its address is
unofficially known to be FS:[0x34] for NT family). FormatMessage
function is used to retrieve a textual error
description. Interpretation of error codes is incompatible with CRT
errno. The logic used by Win32 API for updating it is also different:
it is used to return supplementary information for successfull calls,
so any Win32 function will modify it unconditionally, resetting it to
ERROR_SUCCESS (0) if nothing else makes sense.

Winsock2 provides WSAGetLastError() and WSASetLastError() functions,
accessing the same place as GetLastError() and SetLastError() on the
NT family. Error codes used by Winsock2 are in a separate range, and
the same FormatMessage() call retrieves a corresponding error message
for Winsock2 errors.

One interesting fact about error code symbolic constants: we have both
WSAEINTR and EINTR, they are different in value, and the places where
we expect to find them are different too.

Calling Conventions

This section is entirely unrelated to I/O, and it's valid for 32-bit
systems only.

On x86 (including i386 and above) different compiler vendors used a
lot of different ways to call a function and to return from it. Two of
them survived (for public interfaces) on Windows systems:

"cdecl", that is used on x86 Unices as well; it's normally the
default for any C compiler;

"stdcall" (WINAPI, PASCAL, __pascal, __stdcall), that is used in
Win32 API functions.

The only noticeable difference reflected in the object code is who
cleans up the stack: it's a caller for stdcall and a callee for cdecl.
Win32 port of SBCL foreign function interface is designed to call
external functions of both kinds without explicit convention
specified: ESP register is saved before callout and restored after
return, so it doesn't matter if a callee adjusted it.

As of SBCL, there are two aspects where calling convention still
matter:

"stdcall" function referenced by C code is turned into a symbol
with a special suffix: @n, where "n" is a decimal number of bytes
occupied by arguments; in the C world, this kind of name mangling
prevents resolving a function with missing or wrong prototype in a
way that would misplace the stack pointer;

while a convention-agnostic callout support is easy,
convention-agnostic callback support is impossible: there is no way
to make a foreign code adjust stack after our callback returns. All
callbacks used by Win32 API are stdcall, and all callbacks
generated by SBCL are currently cdecl.

SBCL Modifications: Present

This section describe major changes to SBCL code in my branch relative
to the upstream tree, occasionally mentioning remaining problems and
plans to solve them.

Input and Output

My current branch of SBCL code provides its own lowio-like wrappers
for read and write operations. Those wrappers are ready to work with
both OVERLAPPED and non-OVERLAPPED handles (in the latter case there
is no way to interrupt a blocking call; in the former case, an
operation is cancelled if an interrupt is received).

Replacement for lowio _open() is written in Lisp. It calls
CreateFile() with FILE_FLAG_OVERLAPPED; it makes sense for
communication devices and named pipes, and does no harm for ordinary
files, as long as native _read and _write are not used.
SetCommTimeout() is called immediately after opening the file: if it's
a communication resource, timeout settings are adjusted, so ReadFile()
will return when there are some data available, not wait until the
whole input buffer is filled (this "short read" semantics is what is
expected by SBCL fd-stream layer).

Winsock socket() function is known to return OVERLAPPED socket handles
by default, and it doesn't hurt synchronous operations. However,
SB-BSD-SOCKETS module was going to some length to ensure
non-OVERLAPPED socket creation, apparently, for no reason at all
(though I understand the fear of unexpected problems with synchronous
I/O if a socket is made OVERLAPPED, it is still unfounded: OVERLAPPED
sockets are documented (and known) to work synchronously as well).

My lowio equivalents wait for I/O completion or for an interrupt, and
return EINTR if the latter has happened first (operation is cancelled
with CancelIo if it happens). Two event objects are created for each
thread: one for I/O completion signalling, another one for interrupt
signalling.

As described above, some handles can't be OVERLAPPED, so it's not the
final solution: some operation are uninterruptible yet.

Concurrent operations with the same handle and direction are not
serialized; for seekable files, they are even non-atomic, so
concurrent writes produce files with undefined content (fixable with a
critical section). The thing I find comforting is that for buffered
FD-STREAMs, SBCL will screw it up anyway.

There is also a replacement (actually, a wrapper) for (UNIX-CLOSE) as
well. The only thing it does, beyond calling _close() for lowio
descriptor, is calling closesocket() if a handle was detected to be a
socket when it was alive. Two things to keep in mind:

closesocket() has to be used if we don't want Winsock2 to leak
resources;

closesocket() call after CloseHandle() on the same handle is valid;
according to Winsock, the handle is still alive; it won't reuse it
for another socket, nor will it complain for closing a closed
handle.

(SB!UNIX:UNIX-LSEEK) is now implemented using _lseeki64() function
from MSVCRT; type declarations are adjusted as well, so
(FILE-POSITION) now works with large files.

See

SB!WIN32:UNIXLIKE-OPEN

SB!WIN32:UNIXLIKE-CLOSE

SB!UNIX:UNIX-READ

SB!UNIX:UNIX-WRITE

win32-os.c: win32_unix_read

win32-os.c: win32_unix_write

O_APPEND, FILE_APPEND_DATA, :IF-EXISTS :APPEND

ANSI Standard for CL specifies :if-exists :append to set file position
to the end of file when it's opened. Common Lisp implementations for
Unix-like platforms traditionally translate it into O_APPEND flag for
the call to open(), and SBCL is not the exception.

It's interesting that O_APPEND semantics is not the same thing as
"position to the end while opening" required by the CL
Standard. O_APPEND positions to the end of file before each write
operation, not when opening the file; modern systems also promise
positioning and writing to be atomic as a whole, with a usual
exception of network filesystems (see Unix Haters Handbook for
details).

MSVCRT interpretation of O_APPEND for lowio descriptors is almost the
same as the Unix one, except that positioning and writing is not
atomic: they are done as two separate calls with no locking around
them (mutex won't help here anyway: even if other thread couldn't step
in between positioning and writing, other process could).

Win32 API CreateFile() function provides an equivalent for O_APPEND
(as part of desired access flags, for some reason). I decided to use
it in SB!WIN32:UNIXLIKE-OPEN if O_APPEND is given (some modification
is probably required here: O_APPEND currently forbids reading).

Conclusion: with my replacement for _open(), O_APPEND gets its
Unix-like semantics, and :if-exists :append is interpreted in a way
closer to other platforms but farther from CL Standard requirements.

See

SB!WIN32:UNIXLIKE-OPEN

LISTEN, FD-LISTEN, HANDLE-LISTEN, WAIT-UNTIL-FD-USABLE

SB!WIN32:HANDLE-LISTEN tries to read communication resource statistics
with ClearCommError. If the call succeeds, COMSTAT structure contains
a count of bytes queued for reading in the system buffer.

Other kinds of objects supported by HANDLE-LISTEN in the original code
base are pipes, consoles and sockets (thank Dmitry Kalyanov for the
latter). Unfortunately, support for console objects is broken: when
there is a keyboard event for some "extended key" in the input buffer,
PeekConsoleInput sees it but ReadFile doesn't (no ANSI or Unicode
character is generated, so there is "nothing" to read and ReadFile
blocks).

SB!SYS:WAIT-UNTIL-FD-USABLE internals (polling for readiness) doesn't
end up in (UNIX-FAST-SELECT) or (UNIX-SELECT). Both -SELECT's simply
don't work on win32; they should be either eliminated or rewritten
(the latter is not easy).

See

SB!WIN32:HANDLE-LISTEN

SB!WIN32:COMM-INPUT-AVAILABLE

* PROBE-FILE internals (filesys.lisp)
Device names like "//./COM5" may be used with CreateFile() and,
consequently, with SB-WIN32:UNIXLIKE-OPEN. However, _stat() and
GetFileAttributes() on such names return an error (file "does not
exist"). My current code falls back to opening a handle with zero
dwDesiredAccess mask, and if this operation succeeds, the file is
regarding as existing. GetFileType() on its handle allows to detect if
it's a directory.

While we can open "//./COM5" now, the solution is far from perfect. If
we probe a named pipe in this way, it will count as a connection
attempt, so if a named pipe expects exactly one client connection, the
secondCreateFile call will fail (instead of really opening the file).

As of named pipes, we shouldn't touch them in any way unless it's
needed (by the way, it's a good policy for other files and even other
platforms). (OPEN) implementation should eventually be redesigned not
to probe file at all before opening it (among other things, it is a
race condition). It seems to be possible for any combination of
arguments to (OPEN).

SLEEP and WITH-TIMEOUT

Waitable timer objects are used instead of a simple call to
Sleep(). Consequently, sleeping threads are now interruptible with
sb-thread:interrupt-thread, and if the interrupt function doesn't
unwind, thread continues to sleep after it returns. Deadline of
interrupted (SLEEP) call is not moved when interrupt occurs.

(SLEEP) implementation for Windows now accepts really big integer
intervals: very long sleep is translated into a loop of many moderate
sleeps. This paragraph also applies to a traditional Sleep()-based
implementation, that is still used for non-threaded builds (I'm going
to use waitable timers in both cases soon).

It turned out to be easy to implement the equivalent of
(UNIX-SETITIMER) for threaded builds with a separate signalling thread
and a designated waitable timer object. I have done it, so
sb-sys:with-timeout now works on threaded win32 builds.

See

CL:SLEEP

SB!WIN32:MICROSLEEP

SB!IMPL::WIN32-ITIMER-SCHEDULE

SB!IMPL::WIN32-ITIMER-CANCEL

SB!IMPL::WIN32-ITIMER-DEINIT

STDCALL Name Mangling For Static Foreign Calls: Gone

After SBCL runtime is built, its symbol table is exported into
sbcl.nm. During all further steps of the build process, foreign symbol
references are resolved to their static addresses listed in sbcl.nm.

Name mangling convention described above is not used in system DLLs
when a symbol is resolved dynamically with LoadLibrary() and
GetProcAddress(): those DLLs export unmangled names. However, both
symbol references created by the C compiler and symbol definitions
provided by the import libraries use mangled names. Each address
listed in sbcl.nm for a foreign function points to a tiny piece of
wrapping code from the import library. Foreign stdcall function names
in sbcl.nm are therefore mangled.

Win32.lisp is full of code like (alien-funcall "Sleep@4" ...). It was
a great distraction for non-interactive development and a great
obstacle for interactive one: mangled name can't be resolved if it's
still dynamic for current interactive session; unmangled name works
interactively but breaks compilation.

The code that parses sbcl.nm was modified to remember both mangled and
unmangled variant of a symbol name, getting rid of this maintainance
hell. This change is already accepted into upstream SBCL tree.

Error Handling

For some functions, both CRT error codes from errno and Win32 error
codes from GetLastError() make sense. That's why I hacked
SB!INT:STRERROR to accept a negative FIXNUM designating a Win32 or
Winsock error. For such error code, my version of STRERROR calls
FormatMessage with its absolute value. This change is mostly cosmetic;
STRERROR was not used and shouldn't be used to detect a type of error
programmatically, only to provide a user-readable message.

SB-BSD-SOCKETS module expected to find useful value in the CRT's errno
variable. I've factored out the (SOCKET-ERRNO) function, that returns
an error code for a socket operation in a platform-specific way:
WSAGetLastError() on Win32, errno on other platforms.

There are some simple error code mappings in my lowio equivalents:
ERROR_BROKEN_PIPE on reading is translated to EOF (0 bytes read, error
code doesn't matter); ERROR_OPERATION_ABORTED, that we get after
CancelIo(), is translated into EINTR (errno value). Unhandled error
codes for read and write operation are turned into EIO (errno value).

(UNIXLIKE-OPEN), if an error occurs, maps appropriate Win32 error
codes to ENOENT or EEXIST (they are not assigned to errno but returned
as a secondary value). For any other error, the secondary value is a
negated GetLastError() result: (STRERROR) will convert it into
readable message (see above).

RUN-PROGRAM, MSWIN-ESCAPE-COMMAND-ARGUMENTS

(SB-EXT:RUN-PROGRAM) function had a problem with shell argument
escaping. On Windows the program's caller is responsible for turning
an array of argument into a plain command-line.

I've added (SB-IMPL::MSWIN-ESCAPE-COMMAND-ARGUMENTS) function that
escape the arguments in such a way that CommandLineToArgvW() will
unescape them back. Old escaping code was naive to the core, handling
only the most basic cases (single-word v. multi-word) and ignoring
weird Windows rules regarding "backslashes before a double quote" and
"backslashes before something other".

SB-BSD-SOCKETS

Sockets are now created with OVERLAPPED flag turned on. When a socket
is wrapped into a lowio descriptor, and this descriptor is used for
reading or writing, blocked operation can be interrupted now.

Winsock2 provides a BSD-style non-blocking mode for sockets, but I
don't know if it's possible to retrieve this setting for a socket
handle. While adding support for (NON-BLOCKING-MODE) accessor, I had
to add a slot containing this flag into a socket class: when we can't
introspect, we should remember.

Calls like select() and WSAEventSelect() reset socket to non-blocking
mode. SB-BSD-SOCKET doesn't use these functions. I believe that if
some external library, like USOCKET, calls one of them, responsibility
to restore the old non-blocking state (if it matters) belongs to that
library.

Blocking calls currently are not interruptible (it's possible, even
without native API, but not done yet).

SB-SIMPLE-STREAMS

Some trivial changes were required to make it use
SB!WIN32:UNIXLIKE-OPEN for files, and some changes of similar trivial
nature are yet to be done (calling UNIXLIKE-CLOSE where
appropriate). SB-SIMPLE-STREAMS currently passes all tests on Windows.

Memory-mapped files seem to work (underlying implementation of mmap()
with MapViewOfFile() under the hood is included in the SBCL
runtime). My version of munmap() is a cheat: it ignores the length
argument, unmapping the whole block mapped by
mmap(). UnmapViewOfFile() can't unmap partially, so we have to live
with it (SB-SIMPLE-STREAMS internals don't use partial unmapping
anyway).

Simple streams test suite proved to be an excellent testbed for my
lowio function replacements: a problem with file positioning and
another problem with UNIXLIKE-OPEN flag interpretation were detected
by it. Both problems were obscure enough to stay undetected in normal
conditions; both solutions caused rewriting and simplification of
underlying code.

A known but overlooked peculiarity of Windows manifested itself (and
required a tiny modification of SB-SIMPLE-STREAMS): normally, an open
file can't be deleted. There is a way to open file so it can be
deleted; however, it should be arranged when opening the file. It
requires an extra access flag, and the whole CreateFile() operation
may be denied when it would succeed without this flag.

I've modified UNIXLIKE-OPEN to take an optional flag designating the
desire to delete an open file later, but I didn't modify
SIMPLE-STREAMS to use it yet.

SBCL Modifications: Future

This section describes my ideas of further SBCL improvements (usually
specific for Win32 platform) and the problems in desperate need for
solution.

Thread-Specific System Objects

Sometimes a thread needs a thread-local system object, allocated once
in a thread lifetime and deallocated in a type-specific manner when
thread exits. Thread-specific events used for I/O completion and
interrupt signalling are an example. They are stored in "struct
thread" now, but a nice alternative is possible.

Given the TLS symbol value implementation used by SBCL, it's easy to
implement the allocation/deallocation protocol described above in
Lisp. Not only the pair of private events: e.g. timer objects used for
(SLEEP) should be allocated the same way, not created and destroyed on
each call to (SLEEP). Per-thread events that are used in the condition
variable implementation are good candidates too (they're now allocated
and destroyed each time a thread starts/completes waiting on a
conditional variable).

Stdcall Callbacks

Alastair Bridgewater made available his implementation of stdcall
convention support for alien callbacks. TODO: check it out, test it,
fix it if it's obsolete, ask people why it's not integrated yet(!)...

Dynamic Extent Callbacks

Each alien-lambda (persistent callback) eats a piece of static
space. Most use cases for callbacks don't require them to be
persistent at all. This use case should better be supported by a
special macro, like (DX-ALIEN-FLET ...); it should use a temporal
system memory segment, or a stack, or even a currently-compiled code
segment... Variants are plenty.

Foreign Thread Callbacks

It seems not to be too hard, but requires some design decisions first:

signalling, representation or real thread assignment?

foreign thread extent (creation and destruction)

transparent or explicit support (should the ordinary callback work
in a foreign thread?)

More Interruptible I/O Operations

non-OVERLAPPED handles (anything inherited, console, anonymous
pipes): mad skills are required for console, but it's possible;
native API can be used for other things to turn on
FILE_SYNCHRONOUS_IO_ALERT (so APC will interrupt synchronous
calls).

Open should Open Once

The problems caused by probing a file before opening it are described
above. (CL:OPEN) implementation can (and should) be rewritten to avoid
multiple underlying CreateFile() calls (NB: _stat() does it as well).

:UCS-2 External Format For Console I/O

Consider using ReadConsoleW and WriteConsoleW for console handles, and
set its external format to :UCS-2 unconditionally.

Communication Device Parameters

For opened communication devices, we can emulate termios, or provide a
cross-platform API on top of #+unix termios #+win32 Win32
Communication functions.

Invalidating SAP Objects When Dumping Core

All SAPs except those that point into Lisp memory spaces should be
either set to zero address or changed to primitive objects with
another widetag. BAD things may happen if a SAP survives dumping and
restart and is used afterwards, and nothing prevents it currently.

Generate Static Function References Automatically

Another maintainability problem with SBCL code: foreign functions
referenced by Lisp code but not C code have to be manually added to
win32-os.c:scratch() (don't know if the same amount of work is
required for ldso-stubs.S; is it maintained manually too?).

Undefined references detected between the first and second genesis
should be added to a separate platform-specific file automatically
(probably after ensuring that the symbols are available in C
libraries).

UNIX-SELECT Imlementation (or Elimination)

We need something waitable for each handle to implement
select(). There is an obscure control code for pipes
(FSCTL_PIPE_ASSIGN_EVENT: works, but unpopular; native API required);
we may also use the fact that 0-sized synchronous read on pipe will
block (and FILE_SYNCHRONOUS_IO_ALERT could make it interruptible).
WSAEventSelect for sockets, some mad technique (too long to describe)
for consoles.

If this idea is abandoned (or until it's implemented) remove lisp-side
calls to select(): they call winsock's select(), passing a non-winsock
fdset made of CRT handles. Usually it simply fails, but something
causes EXCEPTION_ACCESS_VIOLATION.

The ultimate goal is a working (SERVE-EVENT), of course.

FINISH-OUTPUT Doesn't

Adding FlushFileBuffers where appropriate is easy. However, making it
interruptible when it could block is a good idea. FlushFileBuffers
doesn't have this feature.

:IF-EXISTS :APPEND Redesign

An ideal solution would provide atomic appends until the first
explicit FILE-POSITION adjustment; after the adjustment, normal
random-access file I/O semantics should be provided.

This kind of thing will be useful on non-win32 platforms as well.

Don't know if it's possible with only one Win32 file handle (it may be).

SetConsoleCtrlHandler

Redirect control-c event to a foreground session (the same thing that
SIGINT causes on other systems).