This page can be safely skipped by
readers concerned only with ASCII source texts.

Background

Because we do not yet have any personal experience with
non-ASCII Unicode
characters at the time of this writing, we considered specifying that
source text for E 0.9 be restricted
to ASCII, and therefore that all non-ASCII source characters may only
be expressed using Backslash-u Decoding. However,
this would be too great a burden on non-English-based programmers wishing
to use E. We obtain a very
similar effect indirectly.

The Java programming language specifies a standard way
of transforming a program written in Unicode into ASCII that changes
a program into a form that can be processed by ASCII-based tools. The
transformation involves converting any Unicode escapes in the source
text of the program to ASCII by adding an extra u -- for example, \uxxxx
becomes \uuxxxx -- while simultaneously converting non-ASCII characters
in the source text to a \uxxxx escape containing a single u.

The JLS defines a Unicode escape effectively as

'\\' 'u'+ <hexDigit> <hexDigit> <hexDigit> <hexDigit>

where the total number of slashes (if any) immediately preceding
this sequence is even.

[Spec] The following Bug MUST
be fixed: What if a non-ASCII character occurs immediately after an
odd number of slashes? The above encoding will produce a Unicode escape
sequence immediately following this odd number of slashes, which will
therefore no longer be considered an actual Unicode escape. Is this also
a bug in the JLS?

Written out at one byte per resulting ASCII character, this
encoding form also defines a character
encoding scheme. We call this encoding form/scheme UTF-J2,
since the Unicode escape defined above can only represent a 16 bit (2
byte) code point. The same section of the JLS also defines two ways of
decoding such text back into a sequence of 16 bit code points. The first
reverses the above encoding with no loss of information:

The exact Unicode source can later be restored from this ASCII form
by converting each escape sequence where multiple u's are present to
a sequence of Unicode characters with one fewer u, while simultaneously
converting each escape sequence with a single u to the corresponding
single Unicode character.

The other decoding method simply decodes each Unicode escape into the
Unicode code point it encodes. The first decoding method would be used
to preserve appearance of the source to those using Unicode editors and
mixing Unicode characters with Unicode escape sequences. We call this
first decoding method a UTF-J2 presentational decode, and consider
it no further. The second would be used prior to all other forms of further
processing, which we call simply a UTF-J2 decode.

UTF-J4

To handle any Unicode character, we extend the above scheme by defining
a Unicode escape to be a sequence of characters accepted either by the
above pattern, or:

'\\' 'u'+ '{' '0' 'x' <hexDigit>+ '}'

We call this extended encoding scheme UTF-J4. A UTF-J4 encode,
when generating a Unicode escape for a non-ASCII code point, SHOULD always
use the first form for 16 bit code points, and SHOULD always use the shortest
encoding in the second form for supplemetary characters.

Rationale: Pleasing Regularities

In the second form of Unicode escape, we include the '0' 'x'
prefix so the string between the curlies will appear to be a numeric
literal. This leaves us open to eventually allowing, for example,
a character name to appear between the curlies instead of a hex code
point.

For purposes of specification, we suppose the following functions

utfJ4Encode(CodePoint[]) -> AsciiByte[]

utfJ4Decode(CodePoint[]) -> CodePoint[]

utf8Decode(UTF8Byte[]) -> CodePoint[]

[Src] The octet sequence input to utf8Decode MAY optionally
begin with the UTF-8 BOM sequence: 0xEF 0xBB 0xBF, which
utf8Decode MUST skip.

Since ASCII is the 7-bit byte subset of both UTF-8 code units and
Unicode code points, we consider AsciiByte[] to be a subtype
of both CodePoint[] and UTF8Byte[].

For all sequences of Unicode code points u:
utfJ4Decode(u) == utfJ4Decode(utfJ4Encode(u)) == ... # and so
on, for any number of UTF-J4 encodings prior to the UTF-J4 decoding.

Therefore, given that we're going to do a utfJ4Decode prior
to further processing, we don't care whether our input is the true
source, or is a UTF-J4 encoding of the source. (If we change the spec
below to track source positions on one of the representations prior
to the utfJ4Decode, then these alternatives would no longer
be strictly equivalent, so under some circumstances we would care.)

From Bytes (Octets) to Source Text

[*] When f is a sequence of octets to be decoded into source
(such as the contents of a file), f MUST be double-decoded
as follows:

sourceText == utfJ4Decode(utf8Decode(f))

The double-decode above yields the same result as

utfJ4Decode(utfJ4Encode(utf8Decode(f))).

If f is in ASCII, then utfJ4Decode(f) also yields the
same result.

[Src] When f is a sequence of octets to be decoded into
source, utfJ4Encode(utf8Decode(f))) SHOULD be in Wysiwyg-ASCII
Format.

[Src] When a source language's grammar uses matched brackets to indicate
nesting structure, source text in this language SHOULD use spaces
for indentation to signal this nesting structure accurately to the
human eye. Further, source text SHOULD NOT include any tab characters
at all.

When rendering text in a fixed width font, tab characters SHOULD
be rendered as whitespace extending to the next modulo-8 tab stop.

[Advisor] An advisor therefore SHOULD alert reviewers of violations
of the above Src RECOMMENDATIONS.

Depending on the density of Unicode escape sequences, the UTF-J4
encoding of the source may or may not be adequately readable for a
review. If this format is adequately readable, reviewers are advised
to look at a rendering of this encoding in a font in which ASCII printing
characters may be easily distinguished. For example, the following
are distinct ASCII printing characters, and should each be unambiguously
recognizable:

1l|!oO0`'

If Raven the reviewer is looking at a readable UTF-J4 encoding of
conforming sources in Wysiwyg-ASCII format, in a font in which all
ASCII printing characters are unambiguously recognizable, then Raven
has grounds for some confidence that the appearance of the text encodes
all the meaning of the text as it will be interpreted by a conforming
language processor. Of course, Arthur the author can still write code
that will confuse Raven the reviewer. But we hope we've made it hard
for Arthur to also confuse Raven about whether she's confused. If
Raven knows she's confused, she can simply reject Arthur's code.

Newline Canonicalization

Once we have source text that passes the above checks, the
following transformations are then applied, logically in order, to create
the source text used for lexical analysis:

Line and Column Numbering. Line and
column numbers designate positions in the source text after the above
steps. The first line is line number 1. The first column is column
number 0.

(The JLS also says that newline canonicalization
happens after interpreting Unicode escapes. Is this really true? It seems
silly, but I'd rather follow Java's lead on this than to try reversing
the order. What does Java do about source positions? Does it say anywhere?)

Only BMP Characters

[Src] Following the above double-decode, the source text MUST consist
only of a sequence of Unicode encoded
characters.

[Src] As of E 0.9, source
text MUST contain only BMP
characters, i.e., only those Unicode encoded
characters whose code
points fit within 16 bits. (From this,
it would seem that UCS-2 characters might be what I mean, but I'm
not sure.)

(Is this too strict? Should we say instead only that source text MUST
contain only 16-bit code points and MUST NOT contain surrogate code points?
Should we demote the other RULES to RECOMMENDATIONS? That would seem to
be the minimal restriction needed to satisfies the following issue.)

Rationale: Indecision is the
mother of convention

Unicode has had a complex but understandable history.
As of the Unicode 3.0 standard or so, it was thought that Unicode
could fit all the world's characters into a 16 bit character set.
Based on this, the Java and Python language s defined a "char"
as 16 bits. Java provided good support for handling Unicode, and became
a leading platform for developing Unicode-ready software. Unfortunately,
the Unicode consortium found that 16 bits was too tight, and expanded
Unicode into a 21-bit character set. It was then unclear what to do
about legacy formerly-Unicode-ready libraries. The litmus test is
indexing:
How does one interpret a source position? What is a counting unit
for determining the length of a string? Currently, the dominant approaches
are:

Java further defines and uses "Modified
UTF-8" rather than standard UTF-8. In Java's modified
UTF-8, a supplementary characters is represented by UTF-8 encoding
each of the surrogate code points in the UTF-16 encoding of
the character. This is explicitly forbidden by the Unicode
spec (D36):

Because surrogate code points are not Unicode scalar
values, any UTF-8 byte sequence that would otherwise map to
code points D800..DFFF is ill-formed.

undesignated
code points -- also called reserved or unassigned code points.
These are either noncharacters,
or code points whose interpretation is not yet specified as of
that version of Unicode. The general category of these is "Cn".

private-use
code points -- those whose interpretation will not be specified
by the Unicode consortium. The general category for these is "Co".

A validator MUST therefore statically reject source text containing
code points that are not encodings of BMP characters.

? pragma.syntax("0.8")
? defmakeChar := <import:java.lang.makeCharacter>
? defisBMPChar(codePoint :(0..0x10_FFFF)) :boolean {
> # If it's not in the range 0..0x10_FFFF, then it's not a valid Unicode code point> > if (codePoint > 0xFFFF) {
> # If it's larger than 0xFFFF, then it's a supplemental code point, > # rather than in the BMP> return false
> }
> # If it's in the BMP, then, even in E 0.9, we can convert it to a char.> defch := codePoint.asChar()
> defcat := ch.getCategory()
> > # If it's general category isn't SURROGATE (Cs) or UNASSIGNED (Cn), > # does that mean it must be a Unicode encoded character? > # What about Private Use (Co)?
> return !(["Cs", "Cn", "Co"].contains(cat))
> }
# value: <isBMPChar>

Rationale: Caught in the Web

E is a distributed
programming language. E
code is often mobile code. Therefore, it could be considered like
a kind of web content, even though it is not a kind of markup. For
possible ease of integration with other tools, and to reduce cases,
it would be good to stay within the W3C's character model.

Unless stated otherwise, all text on this page which is either unattributed
or by Mark S. Miller is hereby placed in the public domain.