Thu, 02 Jul 2009

Strings and Buffers

Handling non-ASCII strings in Perl 5 is a real pain, because there are no
real separate types for binary data and text strings. Mostly the operation
provides a context of either binary or string processing, but function like
length don't, so the answer is dependent on internal
representations about wich the programmer should never care.

In the Perl 6 language design we decided not to repeat that mistake. Since
Strings are objects like everything else, it's easy to invent new types. So in
essence we have two types relevant for our discussion, Str and
Buf.

Str

A Str is notionally a sequence of characters, or a text
string. There's no character encoding attached to it, and while it is surely
stored in a specific encoding scheme internally, it's nothing that programmer
cares about.

A Str co-exists on two (at least) two levels, on codepoint and
grapheme level. A codepoint is everything that the Unicode consortium has
assigned a number and a name, like U+0065 LATIN SMALL LETTER E
or U+0300 COMBINING GRAVE ACCENT. A grapheme is either a
codepoint or a sequence of codepoints that are visually represented together,
for example the two codepoints mentioned before would be printed as a single
grapheme è.

The default level is grapheme, because that's closest to how humans usually
think of characters and text. Specific operations can override the default
abstraction level, or it can be adjusted by pragmas like use
codes;.

Buf

Of course you can also handle binary data in Perl 6. Such data is stored in
objects of type Buf. Notionally a Buf is a list of integers of a
fixed size. It has subtypes for common sizes buf8 is a sequence
of unsigned bytes, buf16 and buf32 store unsigned 16
and 23 bit integers.

When you communicate with anything outside of Perl 6, you'll need
Buf objects for that, because files and terminals only understand
byte streams, not character streams.

There are also a different kind of Bufs which enforce a
specific encoding, for example utf8 can only hold byte sequences
which can be interpreted as UTF-8. They are not strictly necessary, but
provide a nice, convenient interface for some operations.

Conversion

Conversion between Str and Buf is called encoding, the other way round is
called decoding. For example "møøse" is a Str, and
"møøse".encode('Latin-1') returns a Buf, more specifically a
buf8.

On the other hand if you read some bytes from a socket and want to treat
the result as a text string, you decode it: my $str =
$buf.decode('UTF-16LE').

Mixing a Str and Buf that doesn't know about its own encoding in an
operation like concatenation or comparison throws an exception, because those
are the conditions where most Perl 5 programs mess up strings beyond all
repair.

IO

If you read the above, maybe you think that printing a Str to standard
output is an error because the string doesn't know its encoding, so it can't
represented as a byte stream. That's only half the truth; the output handle can
also have knowledge about its own encoding.

When you open a file, you can either specify that it's opened as a binary
file, or you specify an encoding. In the former case reading from the handle
returns Bufs, in the latter Strs.

For the sake of convenience a pseudo encoding called Unicode
exists. (Yes, we know that Unicode defines a character repertoire, not a
character encoding). If there's a byte order mark (short BOM) at the start of
the stream, it is used to determine the encoding. If not, a very simple
autodetection is used: If the file is obviously UTF-16LE, UTF-16BE or UTF-32
(detectable by the position of the zero bytes when encoding ASCII characters)
then the detected encoding is used, otherwise as a fallback UTF-8 is
used. This autodetection scheme is the default.

Conclusions

We learned from the experience that cramming too many different semantics
into a single data type is harmful. So now byte streams and text streams have
different data types, and a clean interface for converting back and forth.

The specification is not set in stone so far, and no compiler implements
the Buf type yet, but it is already planned for Rakudo.

Since this is an important topic to me I will continue to nag the
implementors and language designers about it, and write tests to ensure a
solid implementation.