Since release 6.0, Allegro CL has a new streams model using
simple-streams, which have no element-type, but which act
as if they have element type (unsigned-byte
8). The new implementation is described in
streams.htm. This document describes the Gray
Streams implementation in Allegro CL, which was the stream
implementation in releases prior to 6.0. The
Gray Streams implementation is preserved for backward compatibility.

Gray stream code is not typically included in the Allegro CL
image. Load it with (require :streamc).

In general, a Gray stream is created whenever a file is opened (with
open) with an element-type
specified. If open is called
without an element-type specified, a simple-stream is created. If you
have code which does not specify an element-type, and you want a Gray
stream, specify :element-type 'character. Do note
that you cannot assume that system-created streams will be Gray
streams in version 6.1 or later.

String output streams are always simple-streams even though both
make-string-output-stream and
with-output-to-string accept
an element-type keyword argument. Both operators
create simple-streams regardless of whether a value is specified for
the element-type keyword argument or not.

The transition from Gray streams to simple-streams should be easy and
transparent unless you have done extensive stream customization.

The rest of this document is more or less unchanged from release 5.0.1.
It describes the Gray stream implementation.

Streams in Common Lisp have always been first-class objects and the
stream type a first-class type. The inability of user code to
customize or extend stream behavior has been an unfortunate limitation
especially because the capabilities of the Common Lisp reader and
printer can only be obtained through the interface of a
stream. Suppose, for example, one wants to do normal Lisp printing
while performing some simple output character translations. The only
portable way to do this -- that is, without knowing about each
implementation's internal functions -- would be to rewrite the whole
printer from scratch for the single purpose of interposing a
translation function around every call to write-char and
write-string. This is clearly unacceptable. It should be possible to
customize a stream to add such simple behavior without reimplementing
large portions of Common Lisp.

When Common Lisp was first defined the Common Lisp Object System
(CLOS) was not yet conceived. Some portable stream construction
utilities were originally provided by the several "indirect"
stream types - synonym-stream, concatenated-stream, etc. - but these
provide only very limited kinds of customization. The subsequent ANSI
standard for Common Lisp includes CLOS, so that is now the obvious
mechanism to support user-customizable stream types.

Fairly late in the ANSI standardization process there was some
consideration of closifying streams. The most complete proposal was by
David N. Gray, then of Texas Instruments. Gray's proposal was only a
draft and it acknowledged a number of problems and omissions. It was
intended only as a starting point towards a complete specification.
However, Gray and the standardization committee soon decided it best
not to act because the change to the language would be large (causing
a lot of work for implementors) and also because there was little
actual experience with implementations of CLOS streams. Despite the
feeling that it was not yet the time to adopt closified streams into
the language standard, most members of the committee feel that
redefining streams in terms of CLOS would be intrinsically worthwhile,
and experimentation with extensions should be encouraged.

The implementation of streams in Allegro CL from release 4.0 (on Unix)
through release 5.0.1 (on Unix and Windows) is based largely on the
X3J13 issue writeup entitled "STREAM-DEFINITION-BY-USER, Version
1, 22-Mar-89 by David N. Gray." Some of the text in this chapter
is taken from that proposal, but there are numerous additions,
modifications, clarifications, and comments specific to the Allegro
implementation.

An important feature of the Gray proposal is that it is upward
compatible with both the current language standard and earlier
implementations of Allegro CL. Programmers need not know anything
about closified streams unless they actually use its features.

The symbols naming classes and generic functions for the CLOS Gray
stream interface are exported from the excl package.

The existing Common Lisp I/O functions cannot be made generic because
in nearly every case the stream argument is
optional and therefore cannot be specialized. It is therefore
necessary to define new generic functions which are called internally
by the standard Common Lisp functions. In order to make the meaning as
obvious as possible, the names of the generic functions have been
formed by prefixing "stream-" to the corresponding
non-generic function (e.g. terpri and stream-terpri). Note that for all of these
generic functions, the stream argument must be a stream object,
not t or nil.

Having the generic input functions consistently return
:eof at end-of-file, with the higher-level
functions handling the eof-error-p and eof-value
arguments, simplifies the generic function interface and makes it more
efficient by not needing to pass through those arguments. Note that
the functions that use this convention can only return a character or
integer as a stream element, so there is no possibility of
ambiguity.

There is much uncertainty in the industry about how to document an
object-oriented protocol. There are many ways to do it wrong, and it
is far from clear how to do it right. Of course, problems with
documentation can often be ascribed to poor design of the underlying
system itself or else design based on assumptions that are not made
explicit.

For example, the default method for stream-write-string is
defined to do repeated calls to stream-write-char. The
character-translation problem mentioned at the beginning of
Section 1.0 Introduction to Gray streams in Allegro CL above could be implemented
minimally by defining an :around method for
stream-write-char, and everything ought to work. But what if
some stream specialization -- say, the
excl::file-gray-stream classes provided by the
implementation -- optimizes out the repeated calls to
stream-write-char in the interest of efficiency? The Gray
proposal is silent whether this would conform, but it is clear that
independently-written mixin classes will need to know each other's
assumptions about which publicly-specializable generic functions do or
do not call each other's publicly-specializable generic
function. There is also controversy whether there should be a default
method for a particular generic function or whether a method
definition should be required on specialized stream classes. The
difference is important if mixin methods normally do a
call-next-method, because the question is left open whether the
default method should be, should not be, or may optionally be
shadowed.

In attempting to answer these questions over the years, we have
decided that this is not the best approach, and have opted to redesign
streams per documentation in streams.htm.

This said, there are several kinds of information this document needs
to specify:

The public superclasses (base and mixin) that may be used to define customized stream
classes.

The generic functions that are defined on these classes, when and how they are called,
and which ones call others.

The methods that are defined on these generic functions by the implementation, and what
functionality is required if the user chooses to override any builtin methods.

Which builtin methods and built-in nongeneric functions call other public generic
functions.

Which methods are required of any specialization of a class that are not provided by
default methods on base classes.

Which public superclasses (if any) are required to be mixed into a user-customized
class.

It is a legitimate criticism that this chapter does not exhaustively
cover all these details. The legitimate excuse is that cogent,
defensible design decisions have not yet been made for all of them. In
particular, there is something of a trade-off between (5) and (6)
depending on whether one believes subclassing should be defined in
terms of class inheritance or in terms of generic function behavior
protocol. The new implementation described in
streams.htm addresses these issues in a different
way.

Two kinds of classes are mentioned here. Some are "mixin"
classes intended to be used as super classes of user-defined stream
classes. They are not intended to be directly instantiated; they
primarily provide places to hang default methods. Others are classes
actually instantiated by Allegro CL, for example, to service a call to
open.

Those classes that are sufficiently complete to be meaningfully
instantiated are labeled as Instantiable Class in their
description pages while mixin classes are labeled simply as
Class. You can, of course, further subclass both kinds of
classes. Although most stream classes have their own description page,
the pages do not contain more information than is present in this
document and so we have not provided links.

A subclass of fundamental-stream. Its
inclusion causes input-stream-p to return true. Note: any user-defined stream class that
will do input must include this class. Bidirectional streams may be formed by including
subclasses of both fundamental-output-stream and fundamental-input-stream.

fundamental-output-stream

A subclass of fundamental-stream. Its
inclusion causes output-stream-p to return true. Note: any user-defined
stream class that will do output must include this class. Bidirectional streams may be
formed by including subclasses of both fundamental-output-stream and fundamental-input-stream.

fundamental-character-stream

A subclass of fundamental-stream. It
provides a method for stream-element-type which returns character.

fundamental-binary-stream

A subclass of fundamental-stream. The
Allegro CL implementation requires the :element-type keyword be provided to make-instance
for streams of this class.

fundamental-character-input-stream

Includes fundamental-input-stream and fundamental-character-stream.
Any user-defined stream class that will be used as an argument to read and friends
must include this class.

fundamental-character-output-stream

Includes fundamental-input-stream and fundamental-character-stream.
Any user-defined stream class that will be used as an argument to print, format,
etc. must include this class.

Since in Allegro CL's implementation stream is a CLOS class (more
precisely, a subclass of clos:standard-object) then
the subtypes of stream must
also be CLOS classes.

The stream types other than file-stream and string-stream are sometimes called indirect
streams. The creator functions for these stream types
(make-synonym-stream, make-echo-stream,
make-broadcast-stream, make-concatenated-stream, and
make-two-way-stream) are unmodified from the standard Common
Lisp definitions. These stream types represent an early (and awkward)
attempt in Common Lisp to obtain part of an extensible stream
facility. These stream types may not be defined in an image but the
module defining them, streama, will be loaded if it is
require'd or if (find-class 'x)
is evaluated or make-x is called (where
x is one of the classes). streama is also loaded
automatically if any of the Common Lisp slot-readers for one of these
streams is called. (streama stands for
stream-ansi. The other module is
streamc, which originally stood for
stream-clos, but that interpretation is no longer appropriate
since yet another stream module, streamd, is also
CLOS-based (so `c' and `d' are single letter file indentifiers
only). streamd implements simple-streams.
streamc is only loaded into a Lisp when gray
streams are used. Gray streams include the indirect streams --
synonym-stream, concatenated-stream, etc.)

The generic function approach does not integrate very well with
indirect streams because the indirect stream classes cannot anticipate
the full set of generic functions that user code may want to define
over all streams in order to indirect them to the contained
streams. It is arguable that no purely automatic mechanism can handle
this, not even by defining a general method for
no-applicable-method. For example, consider the problem a
two-way-stream has choosing whether to pass to its input stream,
output stream, or both a call to a generic function about which it
knows nothing. For this reason there has been no attempt to extend or
make customizable the five indirect streams.

The five indirect stream classes are instantiable, but at present
cannot successfully be instantiated other than with their standard
constructor functions (e.g. make-synonym-stream).

The excl::file-gray-stream class is customizable
since it embodies the interface to the file
system. excl::file-gray-stream is a mixin not intended to be
instantiated directly. Its subclasses are normally instantiated by the
open function (see below) although it is also possible to
create them directly with make-instance.

Table of instantiable stream classes

Class

Notes

cl:echo-stream

See discussion above. Created with cl:make-echo-stream.

cl:concatenated-stream

See discussion above. Created with cl:make-concatenated-stream.

cl:two-way-stream

See discussion above. Created with cl:make-two-way-stream.

cl:broadcast-stream

See discussion above. Created with cl:make-broadcast-stream.

cl:synonym-stream.

See discussion above. Created with cl:make-synonym-stream.

input-terminal-stream

All six classes implement streams intended
to support connection to sockets and input-output
devices that connect to a
"stream" of data rather than a fixed file
stored in a file system. A normal call
to open without the :class
argument extension always creates a file
stream. The initial value of
*terminal-io* at startup is a
bidirectional-terminal-stream
stream. One essential difference between these
two groups is evident for the bidirectional
versions. The input and output sides of a bidirectional
file stream access the same data
storage (e.g. on a disk) and so need to cooperate closely
about buffering when input and
output operations are interleaved. They also share a single
file-position pointer. The
input and output sides of a socket stream are completely
separate channels, and socket
streams don't support file position at all.

These classes all require an :fn-in
and/or :fn-out initialization argument
which should be a small integer that
is a Unix file descriptor. (On Windows there is an
internal hidden translation between the
small integers and actual Windows "handles".)
An error will be signaled if the
required initializer is omitted. The input and output
file descriptor of a bidirectional
stream may be and typically are the same.

A character input stream class can be defined by including
fundamental-character-input-stream and defining methods
for the generic functions below. The builtin instantiable classes in
Allegro CL already have appropriate methods, of course.

A binary stream class can be defined by including either or both of
fundamental-binary-input-stream
and
fundamental-binary-output-stream,
and methods for one or
both of the following generic functions.
If you create a binary stream
other than by using open you must also include a keyword
initialization argument for :element-type.

In the public comments on the first Draft Proposed ANSI Standard
several reviewers noted that except for the write-string
function, the language provided no means to request efficient
input-output on large blocks of data. The ANSI standard consequently
added two new functions to the language
read-sequence
and
write-sequence.

Allegro CL also defines generic function versions of these
functions. The nongeneric versions are implemented by calling these
generic functions. Methods are defined for these functions that handle
all legal calls, but not all legal calls will have highly efficient
execution. The existing methods will handle vector sequence arguments
efficiently, and will also handle start and end arguments
efficiently. The Allegro CL implementation will also accept and
efficiently transfer data to and from higher-dimension arrays in the
usual row-major order, although this is an extension to the language
and not defined by ANSI Common Lisp.

The definition of these functions does not permit the useful ability
to operate on sequence elements different from the stream element
type. Such capability would clearly be desirable. (It would, for
example, permit floats to be transferred efficiently, and allow an
application needing to store several different successive sequences of
different types to write them to a single stream and later reread the
data.) However, the behavior of any such operation would depend on bit
representations of objects in an implementation- and machine-dependent
way, and would be incompatible with the strict definition of
read-sequence and write-sequence,
which require automatic conversion of data to the correct element type
of the stream or sequence. Any such capability would therefore need to
be implemented by a different pair of functions.

Streams of arbitrary class may be constructed by an explicit call to
make-instance, but the Gray proposal did not address
how to customize the stream created by open. We
define a simple interface here. The Gray proposal also omits mention
of constructor functions such as
make-string-input-stream,
make-string-output-stream, and their associated
macros such as with-output-to-string. However, there
is nothing these various operators do that can't be performed
explicitly by user code including a call to
make-instance. Unfortunately, Allegro CL's current
string-gray-stream subclasses do not (reliably) support
specialization or even independent instantiation by
make-instance. This is a bug in that some required
initialization is performed by the
make-string-...-stream function.

The open function has been
extended to take a class keyword argument. open passes this
argument to make-instance when it creates the stream,
and as with make-instance, the argument may be a
stream class object or a symbol naming such a class. If the
class argument is not supplied or is nil, open selects one of the
following built-in classes according to the direction and
element-type arguments:

Although the excl::file-gray-stream subclasses
returned by open are all instantiable, at present
they require hidden initialization (for element-type upgrading, buffer
allocation, etc.) and therefore they should only be created using
open. It is fine to further specialize them, but you
are required to create instances of your specializations of these
stream classes using the class keyword argument to
open rather than by calling
make-instance yourself.

On Unix platforms, open will select a terminal-stream
subclass rather than a excl::file-gray-stream
subclass if it detects the argument file is a character special device
or a named pipe. See socket.htm for more
information about creating sockets in Allegro CL.

The original proposal
allowed but did not require
these three existing predicates to be implemented as generic
functions. In Allegro CL,
streamp is not a generic function in order not to impact
speed of opencoded type
dispatching, but the other two functions are made generic.
Normally, the default methods
provided by classes fundamental-input-stream and
fundamental-output-stream are sufficient.

The following function is used by the pretty printer to determine the
output width to be used while pretty printing. Actually, the pretty
printer uses the first of the following three values that return true.