Background

I18N under Python and wxPython

wxPython and Python support the gettext system for I18N (internationalization1).

GNU gettext system

Internationalization of software is supported by the GNU Translation Project. Although the project focuses on the goal of translating software user interfaces in as many natural languages as possible and does not impose a tool set, it recommends the use of gettext (and you should use gettext with Python and wxPython).

Idea behind gettext

The general idea behind gettext is that you write your source code in English and all the natural language strings are also written in English. The strings are inserted inside the source code, you do not use string resource identifiers. But all strings that must be translatable are surrounded a small macro call: _(). The gettext system supports a large set of programming languages including C, C++ and Python.

In C or C++ _() is a special macro that calls on gettext function macro is replaced by the preprocessor (in C or C++) by a call to a gettext function that will search for the proper string at run time using the original English string as the key to the language dictionary that is currently active. If the active language is English, no translation is performed, if the active language is something else the translation is performed if a matching string is found. In Python the _() function is either mapped explicitly to gettext.gettext() by your application or installed by the gettext class API.

The dictionaries are compiled files (files with the .mo extension2). Now, to create the .mo files, you first parse all of your source code with the gettext tools and they generate a .pot file (.POT stands for Portable Object Template) which is a simply formatted text file that contains all of the English strings that must be translated. Each English string acts as the message identifier for that string. Below each English string is a spot for the translated version of that string. You copy the .pot file into a .po 3 and give it to a human translator. Then you compile the translated .po file into a .mo file that you place inside one of the LC_MESSAGE directories of your system. These directories are named after the natural language they refer to. The natural languages supported are the ISO 639 language codes. These codes are a set of two-character language codes.

Applications normally place the various .mo files inside language target specific sub-directories of the directory ./local . The following directory tree show a directory tree for English (en), French (fr) and Spanish (es) would look like.

The .mo files for each language is stored inside the LC_MESSAGES sub-directory under the language code directory. For example, the .mo dictionary file is located inside ./locale/es/LC_MESSAGES.

gettext tools

The GNU gettext tools are available for all OS supported by GNU. The following tools are console tools for the Win32 platform. The GNU tool package include the console programs listed in the following table.

Program

Description

gettext.exe

Displays native language translation of a textual message.

iconv.exe

Decodes text from one encoding to another (e.g. KOI8-R to CP1251)

msgattrib.exe

Filters the messages of a translation catalog according to their attributes, and manipulates the attributes.

msgcat.exe

Concatenates and merges the specified PO files.

msgcmp.exe

Compare two Uniforum style .po files to check that both contain the same set of msgid strings.

msgcomm.exe

Find messages which are common to two or more of the specified PO files.

msgconv.exe

Converts a translation catalog to a different character encoding.

msgen.exe

Creates an English translation catalog. The input file is the last created English PO file, or a PO Template file (generally created by xgettext). Untranslated entries are assigned a translation that is identical to the msgid.

msgexec.exe

Applies a command to all translations of a translation catalog. The COMMAND can be any program that reads a translation from standard input. It is invoked once for each translation. Its output becomes msgexec's output. msgexec's return code is the maximum return code across all invocations.

msgfilter.exe

Applies a filter to all translations of a translation catalog.

msgfmt.exe

Generate binary message catalog from textual translation description.

msggrep.exe

Extracts all messages of a translation catalog that match a given pattern or belong to some given source files.

msginit.exe

Creates a new PO file, initializing the meta information with values from the user's environment.

You should never use a version of gettext older than 0.10.39 (because it produces .po files that cannot be used by the Translation Project without human editing). Version 0.11 is considered stable according to the Translation Project gettext is a set of command line tools and code libraries that have been developed under the GNU umbrella.

gettext file formats

To be written.

Python gettext

The Python gettext module provides internationalization (I18N) and localization (L10N) services for Python modules and applications. It is based on the GNU gettext system.

How to create gettext catalog files

To help creating the gettext binary catalog files, I wrote a Python console program called mki18n.py tha uses the GNU gettext utilities to parse Python source code and create the .po and .mo files. The mki18n.py is used to perform several tasks:

parse all Python source code files of an application and create the .pot file for each target language.

parse all Python source code files of an application and merge the generated .pot with existing .po (files that already contain text translated by human translator).

create the binary .mo files from the .po files.

The mki18n.py module can also be used as an imported module inside other Python programs.

How to write an internationalized wxPython application

The application must contain the following code:

the main module must import the gettext module.

the global code of the main module must call gettext.install() to install the function _() inside the application dictionary. This call identifies the translation domain4 (which normally is the application name), the location of the LC_MESSAGES directories and whether Unicode is used. The following is an example of the call where the dictionary files are under ./locale/xx/LC_MESSAGES (with xx being the language code) and ivcm being the name of the application (and the domain, and the name of the .mo file):

For each supported presentation language, the program must create a gettext. Translation instance by calling gettext.translation(). The arguments to the translation function are the application domain (the name of the .mo file which is often the name of the application), the directory parent of the languages LC_MESSAGES directories and the list of language. For example, if your application is called ivcm and the files are stored under a directory called locales, the support for English, French and Spanish would be set up by doing the following calls:

To activate a the translation of your application for a specified language, the language must be activated by calling the Translation.install() method. For example, to activate the French presentation, you would call:

The gettext.install(domain, localedir, unicode) call instructs the gettext system to look for the dictionary file name built from the components:

localedir/language/LC_MESSAGES/domain.mo

Python and wxPython modules, classes and functions

Python gettext

wxPython wx.Locale class

wxPython wx.GetTranslation()

wxWidgets wxLocale

Controlling the presentation language from the environment variables

The following environment variables control the selection of the translation language. The system uses the language code found in the first environment variable found from the following list:

LANGUAGE

LC_ALL

LC_MESSAGES

LANG

Tools to manage translation dictionary files

I normally use the CRiSP editor to edit .po files and compare several versions of the .po files. There are, however, specialized tools that simplify managing gettext catalog files. These tools are listed here.

Creating .mo files with the [[attachment:mki18n.py]] script

The mki18n.py script helps you create .po and .mo files from your source code files for an application. I describe the process of internationalizing the ivcm application here.

All strings that must be internationalized inside ivcm.py (and its companion files) have the form _("Hello"). All strings inside the source code are normally written in English as this is the convention used by the gettext system.

To use my mki18n.py script, I write a file called app.fil that contains the names of all files inside the application (one file per line, with full or relative path. For example:

Then I run mki18n -p from the directory where ivcm.py is located to parse all source files and create a 'messages.pot' file. The .pot is the original template. You keep this file untouched. If I want to support French then I copy the messages.pot into a .po file named after the domain name (in this case the application name: 'ivcm') and the target language code (in this case: 'fr'). So for French I use the file name: ivcm_fr.po. If I need to support Spanish, I copy messages.pot into ivcm_es.po and so on.

The following lines show a couple of entries inside the non translated ivcm_fr.po:

The next step is to perform the translation. You can use a normal editor to append the French string inside the ivcm_fr.po or use poEdit or any other .po editor. The result of the translation would look like:

Note that every line with a '#' in the first column is a comment or flag. In the example above the python-format flag shows that the strings were extracted from Python source. The #: lines show the line number of the original source.

Some of the flags are set when you re-synchronyze the translations with the source. This resynchronization is required if the source changes after you have created the translated .po file(s).

My mki18n.py script will automatically perform syncronisation if it finds .po files that have the domain_language.po name layout. After a re-synchronization, 'mki18n -p' creates a .new file for every .po file found. In my example, it would create a ivcm_fr.po.new and a ivcm_es.po.new

If the source has not changed, the .new files are equal to the .po file. Otherwise, the .new file contains the new strings to translate, place the string that were removed from the source as comments inside the .po.new file and may also flag some strings as 'fuzzy'. A fuzzy flag indicates that the translation of the original source should probably change because the original string changed. So, I compare the .po and .po.new, and edit whatever is requiered, leaving the finished work inside the .po file.

The final step is to compile the finished .po file into the .mo file.

The .mo file normally reside inside the LC_MESSAGES of a 'locale' sub-directory with a xx/LC_MESSAGES for each supported language:

The ivcm.mo in the fr and sp sub-directories were created mo file by compiling the ivcm_fr.po and ivcm_es.po by using the single command mki18n -m. The ivcm.mo stored inside the en sub-directory was created using the mki18 -e command. The mki18n is a Python scrip that uses the GNU gettext utilities.

The wxstd.mo files contain the compiled string dictionaries for wxPython. These files are distributed with wxPython and are stored under the Python/Lib/site-packages/wxPython/locale/xx/LC_MESSAGES directories (where xx is the language code). Just take them can copy them in your locale directories.

When your application runs, it uses the .mo files identified by your domain (which is ivcm in this case) and the directory (here ./locale) specified by the call gettext.install('ivcm', './locale', unicode=False). The wxPython system uses the wxstd.mo files.

py2exe and gettext

The following gives some sample py2exe setup files which are adapted for gettext .mo files.

Switching between Left to Right and Right to Left Languages

For an application that must be internationalized into languages that read from right to left, such as Hebrew or Arabic, there are other considerations besides translating the strings. The layout of the GUI should be able to switch layout order as well. This is especially important for when options or controls follow a determined direction in the thought process, as having this backwards will confuse the user.

Thankfully, this can be done with rather easily, but the use of sizers is required. You should be using sizers and relative sizes for widgets if you are creating an In8l application anyway, so this shouldn't be a problem.

First we need to determine the language direction. Assume the language variable is a string of the ISO 639-1 code.

Example 2

Ianaré Sévi, 2006/07/04

While the two examples above work flawlessly in Windows, when porting over to Mac OS X and Linux I noticed some problems. After much trial and error here is what should be a more complete solution. You will notice I am getting the language info from the file 'language.ini' - all it contains is an internal code for my application depending on what the user has set using another function. In this way language settings 'stick'. I am using the unicode version of wxPython, and this example has been tested successfully on win NT/2000/XP, Linux (Gnome), and Mac OS X86. The little Linux hack is to get things like calendars and stock buttons to display in the proper language.

- Also, use only ascii characters as your translatable strings (msgid) but use the unicode build of wxPython to display the strings returned by gettext (msgstr). This should allow support for all human languages by your application, however OS must include support as well (ie need to download Windows language pack to correctly view Japanese translation).

Comments

Please feel free to provide any type of feedback on this page or on mki18n.py to me. I will be revamping mki18n.py soon. /Pierre Rouleau.

You can email me at the address provided. I had so much spam at some point, so my email is encoded below.

this is the same way like the python gettext module it does, but more elegant i think...

Example 1 gives me a warning " wxStandardPathsBase::Get﴾﴿: create wxApp before calling this" on line containing "mylocale = wx.Locale(langid)" on Win7. I have a chicken and egg problem. I need this set up before making the wx app. At least I think so.

The word internationalization is often abbreviated as I18N: take the first letter of the word (i) followed by the number of letters in the word (18) and terminate it with the last letter of the word (n) (1)