Are there standard language codes for “World English” and “European English”

Yes! – kind of. It depends on which standard you’re referring to. In any case, there are standard European English and World/International English language codes that can be used for webpages and some software stacks. These implementations include support for date and number formatting information.

The answer to the title question is “en-001” and “en-150”. The rest of this article tries to explain how that came to be and what the consequences of using these language codes in the real world might be.

Language codes are used by authors to indicate the language of a document or other resource. Users use them to signal to software which languages they can understand. Software tries to use them to assist their users in getting language variants of document and resources that they can understand. But where do they come from and what do they mean?

The origins of ‘language codes’

In computer communication and engineering, you’ll often come across standardized two–three letter or numeric codes for countries and a similar set of codes for languages. These two sets of codes are used together to indicate the locale and formatting preference of a document author or of a user.

These codes as used in computer systems today mostly originate from from an international standard known as ISO 3166 Country Codes by the International Organization for Standardization (ISO). These codes are based on the older UN M.49 Standard Country or Area Codes for Statistical Use from the United Nations Statistics Division.

E.g. the language code “en-AU” combines the language code for English with the country code of Australia to form the language Australian English.

ISO 3166 doesn’t include any codes for geographical regions that aren’t nations. The continents and the planet itself thus doesn’t have a standardized code in ISO 3166. If we go back to the source material instead, the M.49, we’ll find a specially designated code “001” which is designated for the geographical region “World” plus other codes for other larger geographical regions such as continents. The ISO 3166 standard has left these region codes unassigned and has avoided any collisions with M.49. I'll get back to these non-nation region codes later.

Where do the language codes used in software come from?

The Unicode Consortium has combined language, country, and region codes with the officially recognized locale and formatting rules for numbers, dates, and more into the Common Locale Data Repository (CLDR). The locale data in the CLDR is sourced from many current internet and international standards. This data repository is in turn sourced by many leading software and operating systems vendors including Adobe, Apple, Google, IBM, Microsoft.

Advertisement

Data from the CLDR repository is sourced directly in to the the popular International Components for Unicode (ICU) library, and is the source data for the locale library in the GNU C Library (glibc). ICU is included with web browsers like Safari, Firefox, and Chrome and is also found in operating systems like macOS and iOS. glibc is found in “everything Linux”.

ICU was developed by IBM since the late 90s, but development has been governed by the Unicode Consortium since mid-2016.

Locale data for many popular language and formatting processing libraries, like Babel for Python and Microsoft .NET, also originate from the CLDR.

Notably, continental and non-nation geographical region variants of any language including English hasn’t been assigned Windows Language Code Identifier (MS-LCID). These languages are thus not recognized in Windows or most Microsoft products. This means that Windows can’t provide locale and formatting information for these types of locales.

Update 2017 April: Windows 10 “Anniversary Update” lets you choose English (Europe), English (World), and several other broader-region locales!

The end of the US locale’s hegemony in CLDR

To keep the CLDR dataset more manageable, many locales inherit data from a similar or familiar locale that makes similar choices for formatting. Most English locales used to inherit from English (US), but in CLDR version 24 this was changed in favor of having all English base languages inherit from either English (World) or English (Europe).

This change significantly reduced the US technological and cultural hegemony over the CLDR dataset. I believe this was a good decision as US companies are often overly focused on the need of the US market at the expense of the international market.

This change in how the CLDR is organized also explicitly added two new language tags into the CLDR: en-001 (International English) and en-150 (European English).

These languages have been technically and conceptually possible to construct using CLDR language and region tags since the beginning of that project. However, their inclusion as explicit datapoints that other languages could inherit rules from meant that CLDR consumer software now would know these two locales as equals to other world languages.

Through CLDR’s efforts to reduce dependency on the en-US locale, they inadvertently also introduced en-001 and en-150 as meaningful options in thousands of software projects.

Although many programs and libraries will recognize locales like en-021 for North-American English, and en-053 for Australian and New Zealand English; they won’t provide locale and formatting data for these languages.

Web browser and software support

Firefox, Google Chrome, and Safari are built using ICU libraries. Recent versions of these browsers, and browsers built upon them, should have full support for at least World English as well as European English. By full support, I mean to say they can be configured to send these language tags in their Accept-Language request headers, as well as support formatting rules using JavaScript functions like (DateObj).toLocaleDateString(), (DateObj).toLocaleTimeString(), and (Number).toLocaleString().

Older versions and other unsupported web browsers will still get a partial match on the en bit of these language tag. E.g. it will match “en” as English even if it can’t recognize “en-150” as European English. The default locale for en is up to the implementation, but you’ll usually find en-US. I expect en-001 will take the place of en-US in at least some implementations in the not so distant future. The algorithm for partial matches is specified in section 3,4 of RFC 4647.

The algorithm that allows for partial matches is also the saving grace for any users, content authors, and programmers who either produce or consume webpages in European or International English. If a web browser or server doesn’t understand or provide content in the preferred language, the fallback language “English (unspecified)” or even a different regional variant like (English Australia) will be returned instead depending on the exact server configuration.

Older versions of Microsoft Internet Information Services (IIS) server doesn’t properly implement the partial match algorithm correctly, and can return unexpected results if the web browser is configured to accept more than four different language tags. I once could have produced a long list of other broken implementations here, but I no longer have any more data on this subject.

It should be pretty safe to use en-001 and en-001 on documents the web. The ICU website lists Google Search as one of their users, which should be reassuring to most skeptics worried about the potential impact of changing their language tags on their search rankings.

What is International and World English anyway?

I left this to the end because I don’t know how to answer it.

Linguistically speaking, I haven’t got the faintest idea of what International English means. Neither languages are formally defined anywhere. But what language is formally defined anywhere? Language is ever changing based on their use. The concept of “International English” can have different meanings to different people.

My own personal definition goes like this: English as used internationally were the speaker or author knows they’re addressing an international audience, and take measures to avoid regional or complicated vocabulary to be better understood by people with different levels of comprehension of English. You could compare my definition to “Simplified English”.

Technically speaking, “English (World)”, or en-001, as defined in CLDR is almost equivalent to “English (United States)”, except for some locale and formatting differences including these:

Uses dd MM YYYY instead of MM dd YYYY date formatting.

First day of the week is Monday instead of Sunday.

Currency symbol is US$ instead of $.

“English (Europe)”, or en-150, as defined in CLDR is equivalent to “English (world)” with the following exceptions:

Uses a comma instead of a period for decimal separator.

Uses a 24-hour instead of 12-hour time format.

Currency symbol is ¤ instead of US$.

For a more complete understanding of what formatting and locale data is included with a given locale, please refer to the CLDR.

Both languages are well suited for users who want one formatting or the other, but who don’t feel like they have any connection to a nation state. Netizens of the interwebs might feel compleled to prefer International English, or maybe European English if they want sensible formatting that follows international standards.