In this article

Creating and supporting OpenType fonts for the Universal Shaping Engine

2/8/2018

26 minutes to read

Contributors

In this article

Microsoft TypographyLast updated: January 2017

This document presents information that will help font developers in creating OpenType fonts for complex scripts included in the Unicode Standard 8.0, but not otherwise supported by a dedicated shaping engine.

Introduction

This document targets developers implementing shaping behavior compatible with the Microsoft OpenType specification for complex scripts not supported by a dedicated shaping engine. It contains information about terminology, font features and behavior of the Universal Shaping Engine (USE). While it does not contain instructions for creating fonts, it will help font developers understand how the Universal shaping engine processes complex script text.

Character classification

The run of text that the shaping engine receives for the purpose of shaping is a sequence of Unicode characters. Itemization preprocessing ensures that the runs of text being shaped contain characters belonging to a single script but may include SCRIPT_COMMON characters. The shaping engine divides the text into syllable clusters and identifies character properties. Character properties are used in parsing syllables and identifying their parts as well as determining whether any special behavior or contextual reordering is required. The Universal Shaping Engine (USE) generates character properties from Unicode data. There is a mapping between Unicode’s categories and the classes used internally by USE. This section defines how USE’s classes and subclasses are derived from Unicode data.

When a decomposition is not defined in UnicodeData.txt, it is up to the font developer to handle any required decomposition during GSUB processing.

Cluster validation, is done based on the decomposed state of a split vowel. Therefore, the validation schemas only take into account the cardinal positions (Pre, Above, Below, Post) since the full decompositions occupy one position only. Therefore cluster validation depends on the sequence of decompositions which may be more restrictive than with ordinary vowels.

Split vowel decomposition needs to be applied recursively so that split vowels get fully decomposed before shaping is applied

Note, if a character belonging to a split vowel class that includes Pre, does not have a canonical decomposition, it is up to the font developer to specify a decomposition. The logical first glyph in that decomposition will be considered to be the VPre. Any subsequent glyphs from that decomposition will not reorder. There are no characters of this category in the currently supported scripts

Note: Font developers must include glyphs for all required decompositions.

Cluster validation

Cluster validation allows sequences of characters to be arranged into groups called “clusters” based on their classification (both class and subclass). In Abugida writing systems, a cluster is an orthographic unit of text that combines multiple phonetic and orthographic elements. It is desirable to control the sequence of characters forming a cluster so that a single visual cluster does not have multiple different encoding sequences as that would create problems for data interchange in terms of stability and security.

USE employs a generalized and permissive cluster structure in order to be flexible enough to accommodate a wide range of script needs. The goal of the clustering logic is to enable what is graphically consistent with a given script’s rules, rather than enforcing particular orthographic or linguistic rules. Such considerations should be applied at another layer, such as a spelling checker.

The maximal cluster scheme used by USE may be visualized as follows:

Visualized form of standard cluster in the Universal Shaping Engine

Schemas and rules for cluster analysis and syllable analysis use the following additional symbols:

X*

sequence of zero or more occurrences of X

X+

sequence of one or more occurrences of X

<X | Y>

disjunction of elements: X or Y

[X]

optional (zero or one) occurrence of X

#

occurrence of a boundary

×

no boundary allowed at indicated position

÷

boundary allowed at indicated position

^

Except

Well-formed character clusters can have combinations of groups as defined below. There are four options:

Independent cluster

< IND | O | Rsv | WJ > [VS]

The independent cluster normally consists of a single member. The only other character class it can combine with is VARIATION_SELECTOR. The BASE_IND, OTHER, Reserved Characters and the word joiner (WJ) can only have a single code point per cluster. When a VARIATION_SELECTOR occurs in any context other than immediately following one of the valid bases (IND, O, Rsv, WJ, B, GB, N, S), it forms an independent cluster.

The only required component of a standard cluster is a BASE or BASE_OTHER. A cluster may optionally begin with a REPH or CONS_WITH_STACKER. A BASE or BASE_OTHER may be followed immediately by a VARIATION_SELECTOR and/or multiple CONS_MOD characters in the order CONS_MOD_ABOVE CONS_MOD_BELOW. Multiple sequences of a HALANT BASE with optional VARIATION_SELECTOR or optional CONS_MOD can occur. The sequence can continue with zero or one CONS_MED for each cardinal position (Pre, Above, Below, Post); zero to many VOWEL characters in each cardinal position; zero to many VOWEL_MODs in each cardinal position; zero to many CONS_FINALs in each of Above, Below, and Post; and lastly, an optional FINAL_MOD.

This is similar to the Standard cluster but terminates in a final HALANT. When a HALANT follows a BASE or BASE_OTHER it will form a cluster. When any character other than a Base follows the Virama there will be a cluster break between the Virama and the following character. Multiple sequences of a HALANT BASE with optional VARIATION_SELECTOR or optional CONS_MOD can occur. A CONS_SUBJ is equivalent to the sequence BASE HALANT.

Number-joiner terminated cluster

N [VS] (HN N [VS])* HN

When HALANT_NUM follows a BASE_NUM it will form a cluster. When any character other than BASE_NUM follows the HALANT_NUM there will be a cluster break between the HALANT_NUM and the following character. A BASE_NUM may be followed immediately by a VARIATION_SELECTOR. Multiple sequences of a HALANT_NUM BASE_NUM with optional VARIATION_SELECTOR can occur.

Numeral cluster

N [VS] (HN N [VS])*

A BASE_NUM may form a cluster with another BASE_NUM when joined using a HALANT_NUM. The join may be repeated. Any BASE_NUM may be followed by a VARIATION_SELECTOR.

Symbol cluster

< S | GB > [VS] (SMAbv)* (SMBlw)*

A SYM character may be followed by an optional VARIATION_SELECTOR and zero to many SYM_MOD_ABOVE, then SYM_MOD_BELOW.

Independent Vowel (IV) plus Dependent Vowel constraints (DV)

The core-specification of the Unicode standard prohibits forming certain IV forms from other bases plus a DV (e.g., TUS Table 12-1). Since these combinations apply to particular pairs and not globally across classes USE maintains a list of prohibited sequences. This list of prohibited sequences is:

Note: The above is a complete listing of prohibited sequences documented in TUS at the time of this writing. This includes sequences from scripts that are not currently handled by USE (e.g., Devanagari).

Combining Grapheme Joiner (CGJ)

CGJ has been omitted from the above schema in order to avoid unnecessary complexity. It may occur anywhere in a cluster with no effect. The purpose of CGJ is to block normalization processing which could change the order of marks in a sequence. CGJ handling will need to be updated if USE is modified to support normalization.

Zero-width non-joiner (ZWNJ)

The zero-width non-joiner is used to prevent a fusion of two characters. It continues a preceding cluster but causes a cluster break after itself when the following character is not a mark character (gc=Mn or gc=Mc). ZWNJ does not reset the cluster model.

Zero-width joiner (ZWJ)

The zero-width joiner is used to fuse two characters. It continues a preceding cluster and joins it to a following character unless the following character is another ZWJ. In which case there will be a cluster break between the two ZWJs. ZWJ does not reset the cluster model.

Standalone characters

Characters of the class BASE_IND occur as standalone characters and do not form clusters with following characters. When a character of the class VS or J occurs outside of a cluster, i.e., at the start of a run, or following one of these standalone characters, they should also be treated as standalone characters.

Defective clusters

When a cluster starts with any character that has UGC=Mc or UGC=Mn, USE inserts a dotted circle glyph (U+25CC) to indicate a broken cluster. Defective clusters do not form extended clusters themselves. A sequence of marks without a valid base forms separate clusters for each mark. Note that an explicit character U+25CC is a valid generic base (GB, BASE_OTHER) and so can form extended clusters.

Maximum cluster length

The maximum cluster length is 31 glyphs. After this limit is reached, a cluster break is forced and a new cluster is started.

OpenType feature application I

Basic cluster formation GSUB

Default glyph pre-processing group

These features are applied together, one cluster at a time. Lookups associated with any of these features will be triggered in the lookup-order specified by the font developer. It is possible to interleave lookups for different features. The order of features given here is recommended.

locl — Localized formsThis provides a starting point to shape any language-specific forms. USE leverages any language system tags provided by the edit control to trigger language-specific substitutions.

ccmp — Glyph composition/decomposition

nukt ― Nukta formsThis feature is intended to enable substitutions with a combining nukta. In general terms, a nukta is a consonant modifier that may be used to indicate an alternate phoneme, often used to indicate foreign sounds to the native language of the writing system.

akhn ― AkhandsUsed to form traditional ligatures from sequences

Reordering group

These features are applied individually in this order, rphf, pref. The output of each of these features is reordered as specified in the next section.

rphf ― Reph formThis feature is traditionally applied to invoke a combining form of a preconsonantal r, which reorders after the following base. This feature may be used generically to identify any component which has the reordering properties of a reph. Feature application is scoped to the first three glyphs in a cluster.

pref — Pre-base formsThis feature may be used generically to identify a post-base component that has the reordering properties of a pre-base medial consonant. The feature is applied to the entire cluster.

Note that reordering is not done until after all of the basic features have been applied.

Orthographic unit shaping group

Like the Default glyph pre-processing group, these features are applied together, one cluster at a time. It is, therefore, up to the font developer to specify the order of lookups for these features in the font’s OTL. The order given here is recommended but not required.

rkrf — Rakar forms

abvf — Above-base formsThis feature should be used to trigger substitutions relating to above-base components.

blwf — Below-base formsThis feature should be used to trigger substitutions relating to below-base components.

half ― Half formsThis feature is used to invoke half forms for scripts that use them.

pstf — Post-base formsThis feature should be used to trigger substitutions relating to post-base components.

vatu ― Vattu variantsThis feature is used to trigger ligatures with a below-base form.

cjct ― Conjunct formsThis feature is used to complete basic conjunct forms not already covered by other features.

Glyph reordering

All reordering and anchoring of marks is done in relation to the base consonant. USE does reordering in a single late phase. This is because all reordering is dependent on the formation of the base which may be modified during basic cluster formation. USE does not do checks or look-ahead in a font’s feature tables. Therefore feature lookups for the basic features must be designed for the logical glyph order before any reordering has been applied. There are two categories of glyphs that reorder: feature based and property based. Actual glyph reordering is done between basic cluster formation (OpenType feature application I) and Topographical substitutions (first part of OpenType feature application II). Reordering is applied in logical order: rphf, pref, VPre, VMPre:

Before reordering

After reordering

When the base cluster is broken by an explicit virama that has not been replaced during basic cluster formation the reordering is impacted since reordering components do not reorder past an explicit virama:

Before reordering

After reordering

If a font developer wants the reordering behavior to not be blocked by an explicit virama, they can substitute the virama for an alternate glyph during basic cluster formation so that USE treats the cluster as a continuous cluster without explicit virama.

Feature-based reordering

Sigla

Name

Description

rphf

Reph form

Many abugida scripts render a preconsonantal r as a sign above the consonant it precedes. This sign is called reph. For the purposes of OT layout, reph is normally rendered with a mark glyph, and as such, must follow the base to which it applies. Contexts which produce the reph glyph must use the feature. The output of this feature reorders after the following full base. Reph does not reorder past an explicit Virama. Lookups under the rphf feature should output no more than one glyph per cluster.

pref

Pre-base form

Scripts such as Javanese and Balinese reorder a medial consonant to the beginning of a cluster based on context. The contextual logic is encoded in the font’s OT logic. Cases which require reordering should use the feature to identify the glyph or glyphs that is or are to be reordered. One or more glyphs may be substituted for a single feature glyph that is to be reordered before the first spacing glyph in the cluster or the first spacing glyph after an explicit virama if present. Only one such glyph is reordered per cluster.

Note that prebase medial consonants (CONS_MED_PRE) do not get reordered automatically by USE. Rather, the font designer is expected to use the feature to signal when a glyph belonging to this class should be reordered through the application of the pref feature. Only one glyph per cluster can be reordered using .

Property-based reordering

Sigla

Name

Description

R

REPHA

Pre-base REPHA is reordered by USE after a following full base as if the feature has been applied. Font developers do not need to use the feature explicitly.

VPre

VOWEL_PRE

Pre-base vowels and pre-base vowel components from split vowels are reordered before the base glyph and, if present, before a pre-base glyph reordered via the feature.

VMPre

VOWEL_MOD_PRE

Pre-base vowel modifiers are reordered before the base glyph and, if present, before a pre-base vowel and/or before a pre-base glyph reordered via the feature.

Note that the category REPHA is not currently supported by USE.

Note that the split vowels, VOWEL_PRE_ABOVE, VOWEL_PRE_ABOVE_POST, and VOWEL_PRE_POST have multiple positions that include a pre-base element. Since split vowel decomposition is done before reordering, only one glyph will have the VOWEL_PRE class, and so only this vowel class needs special handling at this stage.

OpenType feature application II

Topographical features, GSUB

USE applies positional features required for scripts like Arabic which have alternate glyph shapes depending on the position of a glyph within a word.

isol ― Isolated forms

init ― Initial forms

medi ― Medial forms

fina ― Final forms

USE applies these as required features on a per cluster basis in order to invoke a particular form based on non-joining or white space boundaries. Non-joining boundaries occur between glyphs belonging to a joining script when one or both characters have a non-joining property that applies to the side of the connection on which a join would occur. The control characters ZWJ and ZWNJ may be used to artificially invoke or prevent a join. Joining properties are defined by the Unicode property Joining Type in ArabicShaping.txt.

Note:Support for topographical features for non-joining scripts is not currently implemented in USE. Additional specification is required.

Standard typographic presentation, GSUB

The remaining required features are applied all together to the entire run. It is up to the font developer to specify the order of lookups for this set of features:

abvs — Above-base substitutions

blws — Below-base substitutions

calt — Contextual alternates

clig — Contextual ligature

haln ― Virama forms

liga — Standard ligatures [horizontal text only]

pres — Pre-base substitutions

psts — Post-base substitutions

rclt ― Required contextual forms

rlig — Required ligatures [horizontal text only]

vert — Vertical writing [implemented by edit control]

vrt2 — Vertical alternates and rotation [implemented by edit control]The vertical layout features are applied when text layout is vertical and triggered by the edit control as a custom feature. Only one feature vert or vrt2 should be applied, not both.

Custom substitution features requested by the application, GSUB

Positional feature application, GPOS

During GPOS processing, these required features are applied simultaneously to the entire run. The order specified here is recommended, but it is up to font developers to define the order of GPOS lookups for this set of features in OTL.

curs — Cursive positioningThis feature should be used to position bases in relation to other bases.

dist — distancesThis feature should be used to make required width adjustments.

kern — KerningThis feature should be used to provide optimal letter spacing. This feature can be disabled by edit controls.

mark — Mark positioningThis feature should be used to position marks in relation to a base sign.

abvm — Above-base mark positioningThis feature is functionally equivalent to the mark feature. It may be used as an organizing mechanism to separate above mark processing from below mark processing.

blwm — Below-base mark positioningThis feature is functionally equivalent to the mark feature. It may be used as an organizing mechanism to separate below mark processing from above mark processing.

mkmk — Mark to Mark PositioningThis feature should be used to position marks in relation to preceding marks.

An important OT technique for excluding certain base glyphs from contextual lookups is to classify the base glyph as a mark in the font's GDEF table, since marks can be selectively included or omitted from OT processing. However, whenever this technique is used, the width of the base glyph must be added back using the feature. This is necessary because OT processing cancels the width associated with a mark. It is necessary to cancel the width of a non-spacing mark because it is not clear where to apply the width of a non-spacing mark during OpenType processing.

A typical use case of this is Javanese which has prebase vowels. Since the prebase vowels do not reorder until after basic cluster formation, they are present in their logical position. This may interrupt other contextual substitutions. If the vowels are treated as marks, they can be excluded from OT context, and thus reduce the number of contextual rules required for processing. Consequently the expected width of the prebase vowels may be restored with the dist feature, for example (in VOLT OT Language):

Other encoding issues

Handling invalid combining marks

Combining marks and signs that do not occur in conjunction with a valid base are considered invalid. USE treats an invalid mark as a separate cluster and displays the stand-alone mark positioned on a dotted circle (U+25CC). If multiple marks are required to position on a dotted circle, the dotted circle can be explicitly inserted into the text stream followed by any marks in accordance with the standard clustering rules.

To allow for shaping engine implementations that expect to position an invalid mark on a dotted circle, it is recommended that font using USE contain glyphs for the dotted circle character, U+25CC. If this character is not supported in the font, such implementations will display invalid signs on the missing glyph shape (white box).

Recommended Glyphs

Unicode code points that are recommended for inclusion in any font using USE are:

Code point

Description

U+200B

Zero Width Space

U+200C

Zero Width Non-Joiner

U+200D

Zero Width Joiner

U+25CC

Dotted Circle

U+00A0

No-break space

U+00D7

Multiplication sign

U+2012

Figure dash

U+2013

En dash

U+2014

Em dash

U+2015

Horizontal bar

U+2022

Bullet

U+25FB

White medium square

U+25FC

Black medium square

U+25FD

White medium small square

U+25FE

Black medium small square

Appendix

Writing system and language tags

OpenType features are enabled in a font according to both a designated script and language system. The language system tag specifies a typographic convention associated with a language or linguistic subgroup. Not all software applications support specific language tags for use when rendering text runs.

NOTE: It is strongly recommended to include the “dflt” language tag in all OpenType fonts because it defines the basic script handling for a font. The “dflt” language system is used as the default if no other language specific features are defined or if the application does not support that particular language. If the “dflt” tag is not present for the script being used, the font may not work in some applications.

The following tables list the registered tag names for scripts currently supported by USE. This list will grow as new complex scripts are enabled by Unicode and USE is updated. Language system tags are not listed here and should be determined by font developers as appropriate for the script concerned.

Registered tags for the Universal Shaping Engine

Script

Script tag

Balinese

bali

Batak

batk

Brahmi

brah

Buginese

bugi

Buhid

buhd

Chakma

cakm

Cham

cham

Duployan

dupl

Egyptian Hieroglyphs

egyp

Grantha

gran

Hanunoo

hano

Javanese

java

Kaithi

kthi

Kayah Li

kali

Kharoshthi

khar

Khojki

khoj

Khudawadi

sind

Lepcha

lepc

Limbu

limb

Mahajani

mahj

Mandaic

mand

Manichaean

mani

Meitei Mayek

mtei

Modi

modi

Mongolian

mong

N’Ko

nko

Pahawh Hmong

hmng

Phags-pa

phag

Psalter Pahlavi

phlp

Rejang

rjng

Saurashtra

saur

Sharada

shrd

Siddham

sidd

Sinhala

sinh

Sundanese

sund

Syloti Nagri

sylo

Tagalog

tglg

Tagbanwa

tagb

Tai Le

tale

*Tai Tham

lana

Tai Viet

tavt

Takri

takr

Tibetan

tibt

Tifinagh

tfng

Tirhuta

tirh

Note: script tags are case sensitive (script tags should be lowercase) and must contain four characters.