Public Review Issue #96

Allowing Special Characters in Identifiers

Significantly tightened the requirements for ZWJ and ZWNJ by reducing the number of possible scripts, and
simplifying the sequences. Also added the equivalent characters needed for Mongolian.

This PRI affects the use of special characters (ZWJ, ZWNJ and Mongolian separators) in identifiers. It may be
relevant in a variety of contexts, including such areas as international domain
names for Arabic, Persian, Sinhalese, Khmer, and Malayalam. If you believe that there are any other languages requiring the use of special characters, please respond
as directed on http://unicode.org/review and include the PRI number and Revision Number in your message.

The use of format characters in identifiers is problematical
because the formatting effects they represent are normally just stylistic or otherwise out of scope for identifiers. To make matters worse, it's possible to
misapply format characters such that users can create strings that look the same but actually contain different characters, which can create security problems
(see
UTR# 36: Unicode Security Considerations).

For these reasons format characters are normally excluded from
Unicode identifiers. However, visible distinctions created by certain
format characters (particularly the joiner controls) are necessary and
carry meaning in certain languages. A blanket exclusion of format
characters makes it impossible to create identifiers based on
certain words or phrases in those languages. Identifier
systems that attempt to provide more natural representations of terms, such as
geographic names, company names, and so on should consider allowing these
characters, but limited to particular contexts where they are necessary.

The goal for such a restriction of format characters to particular
contexts is to

allow the use of these characters where required in normal text

exclude as many cases as possible where no visible distinction results

be simple enough to be easily implemented with standard mechanisms
such as regular expressions

Normal usage, as meant here, does not include technical usage such as
mathematical expressions or pedagogical use (eg, illustration of
half-forms or joining forms in isolation).

Script Restriction. In each of the following cases, the specified sequence must only consist of characters from a
single script (after ignoring Common and Inherited script characters).

Performance. Parsing identifiers can be a performance-sensitive task. However, these characters are quite
rare in practice, thus the regular expressions (or equivalent processing) only rarely would need
to be invoked. Thus these tests should not add any significant performance cost overall.

The characters and their contexts are given by the following:

A.
ZWNJ in the following contexts:

Breaking a cursive connection. That is, in the context based on
the Arabic Shaping property, consisting of:

A Left-Joining character, followed by zero or more Transparent
characters, followed by a ZWNJ, followed by zero or more Transparent
characters, followed by a Right-Joining character

Example: In Khmer, U+17A2 U+200D(ZWNJ) U+17CA U+17B7 U+17A2 U+17BB U+17CA U+17C7 [អ៊ិអុ៊ះ]
is a case where the first TRIISAP needs to be escaped, but the second does not (as there is a below base vowel).

Example: The Malayalam word for eyewitness. The form without the ZWNJ is incorrect in this case.

Figure 2.

B. ZWJ in the following context:

In a conjunct context. That is, a sequence of the form:

A Letter, followed by a Virama, followed by a ZWJ,
where the Letter and Virama are both in the Sinhala script

Example: The Sinhala word for
the country 'Sri Lanka' in Figure 3A, which
uses both a space character and a ZWJ. Removing the space gives the text
in Figure 3B which is still readable, but removing the ZWJ completely
modifies the appearance of the 'Sri' cluster and gives the text in
Figure 3C.

Figure 3.

C. Mongolian Separators (NNBSP or MVSs) in the following context:

Between Mongolian Letters. That is, a sequence of the form:

A Mongolian Letter, followed by NNBSP or a MVS, followed by a Mongolian Letter.

Comparison Cases

The above description restricts the usage of Joiner and Nonjoiner quite substantially from Revision 1 of this Public Review Issue. This restriction was based on a review of the cases where these
characters would be required for semantic differences relevant to identifiers. The other specified cases of Joiner or Nonjoiner usage in the Unicode
Standard were not considered to be required for identifiers. They are listed here for comparison, so that reviewers can look
over these cases to see if there are good reasons for including them in the above list.

Non-Semantic

Cases that do not carry semantic differences (or at least differences which are not sufficient to be required in identifiers for modern languages):