Rationale and requirements for the localized Date Handler module for GRAMPS.

−

==Why have different date handlers?==

==Why have different date handlers?==

Different cultures and regions tend to have specific and different conventions for parsing and displaying dates. For example, the month and day order is different between most european coutries and the US. Also, each language has its own set of acceptable modifiers and qualifiers for the date: things like "from X to Y" or "between X and Y" may have different word order in different languages. Same with "around", "calculated", "estimated". Add to this calendar names and you have a compelling need for a dedicated date handler.

Different cultures and regions tend to have specific and different conventions for parsing and displaying dates. For example, the month and day order is different between most european coutries and the US. Also, each language has its own set of acceptable modifiers and qualifiers for the date: things like "from X to Y" or "between X and Y" may have different word order in different languages. Same with "around", "calculated", "estimated". Add to this calendar names and you have a compelling need for a dedicated date handler.

By providing date handlers, several problems are resolved.

By providing date handlers, several problems are resolved.

−

* Dates entered by users of that language in almost any form can be parsed by GRAMPS.

+

* Dates entered by users of that language in almost any form can be parsed by Gramps.

* The displayed dates will look clear and correct to the users.

* The displayed dates will look clear and correct to the users.

* Translators do not have to worry about translating regular expressions like <code>(from|before|between)</code>.

* Translators do not have to worry about translating regular expressions like <code>(from|before|between)</code>.

+

+

Unfortunately, it turned out that the maintenance burden of date handlers across all the languages is very high compared to the regular maintenance overhead of strings in the translation database. Under {{bug|6926}} we attempted to shift the balance back from coding to translation database. The new system also allows to support inflections for month names in the languages where it is needed. Date handler code still exists, but is much simpler and easier to maintain. See also [[#Known bugs|the related open issues]].

==How to write a date handler==

==How to write a date handler==

−

The framework for date handler plugins is in place. Here are the rules the language-specific plugins must obey to be compatible with the framework:

+

{{man warn|Since new code for Gramps is developed on trunk, the present instructions only refer to the current API. If you are interested in older API documentation, see the revision history on the wiki and in the [[Getting_started_with_Gramps_development#Get_the_source_tree|source control]].}}

−

* The plugin must define '''two''' classes: one for parsing and one for displaying dates.

+

Please use the generated documentation on the [http://www.gramps-project.org/docs/date.html date handler framework] as your programming reference. Here are the some highlights on how a date handler plugin for a new language should be developed.

−

* The parser class must derive from the [http://www.gramps-project.org/devdoc/api/2.2/private/DateHandler._DateParser.DateParser-class.html DateParser] class:

+

* The plugin must translate some strings in the translation database. Special conventions allow for finer linguistic control for inflected forms of month names for languages where this is needed.

+

* The plugin must define '''two''' classes: one for parsing and one for displaying dates. Various degrees of customizations exist, depending on how different the standards for formatting the dates in your locale are from English.

+

* The plugin should be registered in the system.

+

* Before submitting it to the code base, it should be tested using the existing testing framework.

+

+

===Localizing the date strings===

+

Use the regular [[Translating Gramps|translation routine]] to translate the strings such as month names, date modifiers etc. These strings are stored in the class '''DateStrings''', which is used both from the date parser and the date displayer.

+

+

If your language has special inflection rules, note that these strings are processed with [http://www.gramps-project.org/docs/gen/gen_utils.html#gramps.gen.utils.grampslocale.GrampsTranslations.lexgettext gen.utils.grampslocale.GrampsTranslation.lexgettext] into instances of [http://www.gramps-project.org/docs/gen/gen_utils.html#gramps.gen.utils.grampslocale.Lexeme gen.utils.grampslocale.Lexeme]. Use the examples in the Lexeme docs to learn how to provide all the inflections you need for all the contexts used later in the date displayer.

+

+

For example (from ru.po):

+

+

#: ../gramps/gen/datehandler/_datestrings.py:70

+

msgid "localized lexeme inflections||May"

+

msgstr "И=май|Р=мая|Т=маем|П=мае"

+

+

All the provided inflections and all their unambiguous prefixes will also be automatically recognized as valid synonyms for the date parser base class code.

+

+

===Providing a localized date displayer class===

+

The displayer class must derive from the '''DateDisplay''' class:

+

from _DateDisplay import DateDisplay

+

class MyDateParser(DateDisplay):

+

...

+

+

The displayer class must provide <code>display()</code> method.

+

+

If you inherit the default implementation, then all the dates will be displayed in iso style, without any visible difference between span and range dates.

+

+

The next level of customization is to select another pre-canned alternative, provided in the base class, as follows (like DateDisplayEN does):

+

+

display = DateDisplay.display_formatted

+

+

This will be sufficient for over half of the currently supported languages, that don't tweak the list of formats available for date display.

+

+

The core of display_formatted displays both regular and compound dates, and adorns them, if necessary, with modifiers (as in "before..."), quality indicators (as in "calculated..."), and calendar/non-standard year start information.

+

+

If your language requires inflections of month names dependent on modifiers, or, for instance, whether the date is a start or a stop date in a range, there are special strings you need to translate to indicate the appropriate inflection. Documentation for this advanced scenario can be found in the source code comments to DateDisplay.

+

+

If your language overrides Display.formats to something else, it should also override _display_calendar (or, separately, the _display_gregorian/_display_hebrew/... whatever methods that delegate to _display_calendar by default with the appropriate month lists). The override of _display_calendar should format the date according to the currently selected custom format, using self.format number as an index into the self.formats list.

+

+

===Providing a localized date parser class===

+

The parser class must derive from the '''DateParser''' class:

from _DateParser import DateParser

from _DateParser import DateParser

class MyDateParser(DateParser):

class MyDateParser(DateParser):

...

...

−

::The parser class must provide <code>parse()</code> method. In fact, since the base class already defines such method, it is most likely that you will only need to re-define class constants and, maybe, the <code>init_string()</code> method.

+

::The parser class must provide <code>parse()</code> method. In fact, since the base class already defines such method, it is most likely that you will only need to re-define class constants and, maybe, the <code>init_strings()</code> method.

−

* The displayer class must derive from the [http://www.gramps-project.org/devdoc/api/2.2/private/DateHandler._DateDisplay.DateDisplay-class.html DateDisplay] class:

+

===Localizing the date formats===

−

from _DateDisplay import DateDisplay

+

If you use DateDisplay._display_calendar in your date displayer, whether directly or via DateDisplay.display_formatted, then your dates will be formatted using the translated format strings therein. The month names (whether short or long) in these formats are used to format an instance of the class Lexeme, initialized to the set of inflection forms from the translated month string.

−

class MyDateParser(DateDisplay):

+

+

These formats all have a warning in the translation database sending you to the entrance point to these docs. You have the following options in translating such a format string:

+

+

1) If you don't translate the string, it will just use the same order as in English, without any inflection selection.

+

+

2) If you omit the inflection selection in a format,

+

then the default inflection will be used, or, if your language has no inflections, just the one and only name for the month (see the documentation for class Lexeme for details). '''REMEMBER''' the format specifiers themselves must not be translated, otherwise there will be a runtime error in the formatting code. E.g.:

+

+

msgid "{day:d} {long_month} {year}"

+

msgstr "{day:d}/{long_month}/{year}"

+

+

(Note how the words "long_month" remain in English!!!)

+

+

3) However, if your language does support inflections, here you should select the appropriate inflection for the month name,

+

'''using the same name of the inflection as you entered in the translated month string'''. Again, see the documentation for class Lexeme for details. E.g.:

Some of the strings need not be translated at all for your language if you don't use the inflection control feature of _display_calendar, e.g.:

+

+

#. first date in a span

+

#. You only need to translate this string if you translate one of the

+

#. inflect=_("...") with "from"

+

#: ../gramps/gen/datehandler/_datedisplay.py:160

+

msgid "from|{long_month} {year}"

+

msgstr ""

+

+

Documentation for this advanced scenario can be found in the source code comments to DateDisplay.

+

+

===Upgrading an older date handler to the framework from {{bug|6926}}===

+

First, before changing your code, run the helper code to generate a snippet you can merge into your language's xx.po. See [http://www.gramps-project.org/docs/date.html#main the documentation].

+

+

If you stop here, the handler should work already (strings previously in _grampslocale.py are now picked up from the translation database, and the rest remains the same). However, it'll likely have lots of redundant code and strings in it...

+

+

Then remove from your DateDisplayXX class any overrides of date strings from base class, such as

Here we unite the alternative spellings Nisan and Nissan into the same lexeme.

+

+

Alternatively, you can extend the base string table that is initialized from the date strings explicitly from the code, as before. For example, here's how the Russian parser adds an alternative name for the Persian calendar:

+

+

def init_strings(self):

+

DateParser.init_strings(self)

+

DateParser.calendar_to_int.update({

+

'персидский' : Date.CAL_PERSIAN,

+

'п' : Date.CAL_PERSIAN,

+

})

...

...

−

::The displayer class mus provide <code>display()</code> method. Again, the base class already provides such method, but you may need to redefine it for your language, it is not that hard.

−

* The module must register itself as the date handler. This is done by inserting the following code at the end of your <code>_Date_xx.py</code> file:

+

====If you don't have custom language-specific formats====

+

If you override ''formats'' and it corresponds exactly to the formats in the DateDisplay.formats, only changing the strings inside to translate them to your language, then remove the formats override and any overrides of ''_display_gregorian''/''_display_julian''/... and ''display'' methods.

+

+

Just alias your ''display'' to ''DateDisplay.display_formatted''.

+

+

====If you have custom language-specific formats====

+

Consider rewriting your existing overrides of _display_gregorian and display to a single override of _display_calendar.

+

At the very least, you'll need to update the signature of your _display_* methods, to match _display_calendar. You may just add (and ignore) a trailing **kwargs if you're lazy...

+

+

====Converted handlers====

+

Add yours here when done!!!

+

* FR (romjerome)

+

* HR (josip)

+

* RU (vassilii)

+

+

===Registering the localized date handler===

+

The module must register itself as the date handler. This is done by inserting the following code at the end of your <code>_Date_xx.py</code> file:

from _DateHandler import register_datehandler

from _DateHandler import register_datehandler

register_datehandler(('xx_XX','xx','xxxxx','xx_YY'),

register_datehandler(('xx_XX','xx','xxxxx','xx_YY'),

Line 32:

Line 154:

::where <code>MyDateParser</code> and <code>MyDateDisplay</code> are the classes defined for parsing and displaying, and the items in quotes are language identifiers that '''may possibly''' be associated with your language. For example, different systems use <code>ru</code>, <code>RU</code>, <code>ru_RU</code>, <code>koi8r</code>, <code>ru_koi8r</code>, <code>russian</code>, <code>Russian</code>, <code>ru_RU.koi8r</code>, etc. to identify the Russian language.

::where <code>MyDateParser</code> and <code>MyDateDisplay</code> are the classes defined for parsing and displaying, and the items in quotes are language identifiers that '''may possibly''' be associated with your language. For example, different systems use <code>ru</code>, <code>RU</code>, <code>ru_RU</code>, <code>koi8r</code>, <code>ru_koi8r</code>, <code>russian</code>, <code>Russian</code>, <code>ru_RU.koi8r</code>, etc. to identify the Russian language.

−

Then you need to import the parser to GRAMPS by adding

+

Then you need to import the parser to Gramps by adding

import _Date_xx

import _Date_xx

line to __init__.py file located in the same directory.

line to __init__.py file located in the same directory.

−

That's it for the requirements. The example date handling plugins can be found in <code>_Date_xx.py</code> under the <code>src/DateHandler</code> directory.

+

That's it for the requirements. The example date handling plugins can be found in <code>_Date_xx.py</code> under the <code>gramps/gen/datehandler</code> directory.

+

+

===Classes and relations===

+

{{man warn|not updated yet -- any way to ask Sphinx to do it?}}

+

A quick overview of current [[Media:Date classes.svg|classes]] and [[Media:Date packages.svg|relations]] between Date handlers.

It is possible to add new one, like Chinese calendar, Indian Civil calendar, Bahá'í calendar, Mayan calendar, Zoroastrian calendar or Japonese calendar ... into gen/lib/calendar.

+

It is possible to add new one, like Chinese calendar, Indian Civil calendar, Bahá'í calendar, Mayan calendar, Zoroastrian calendar or Japanese calendar ... into gen/lib/calendar.

+

We need to have rules which properly convert this new calendar to sdn (serial date number) and from sdn to calendar.

We need to have rules which properly convert this new calendar to sdn (serial date number) and from sdn to calendar.

==How to test a date handler for your locale==

==How to test a date handler for your locale==

−

===Testing into GRAMPS===

+

===Testing with Gramps===

+

When you [[Running_a_development_version_of_Gramps|run from the Git tree]], you have the [http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/plugins/tool/dateparserdisplaytest.py dateparserdisplaytest.py] debug tool available under {{Man menu|Menu -> Tools -> Debug ->}}.

−

You could have a look at [http://gramps.svn.sourceforge.net/viewvc/gramps/trunk/src/plugins/tool/DateParserDisplayTest.py DateParserDisplayTest] debug tool on SVN and available under '''Menu -> Tools -> Debug ->'''.

# Compare source and target dates in the entries on [[Gramps_4.0_Wiki_Manual_-_Main_Window#Events_Category|Events View]]. Alternatively, compare the birth and death dates in the People View. In the latter view, the failed tests are marked with a "Fail" tag on the person, you can use the "Customize view" option to add the "Tags" column to sort on the pass/fail status.

* There is an other debug test under [http://gramps.svn.sourceforge.net/viewvc/gramps/trunk/src/gen/lib/test/date_test.py gen.lib] (see [[Unit_Test_Quickstart#Using_Your_Unit_Test|Unit test]]), but it returns false positive/negative with modifier, calendar, quality, span, range dates, because it does not use translated words!

+

See also [[Testing Gramps]].

−

But you can try to generate a localized testing module by translating english words:

Why have different date handlers?

Different cultures and regions tend to have specific and different conventions for parsing and displaying dates. For example, the month and day order is different between most european coutries and the US. Also, each language has its own set of acceptable modifiers and qualifiers for the date: things like "from X to Y" or "between X and Y" may have different word order in different languages. Same with "around", "calculated", "estimated". Add to this calendar names and you have a compelling need for a dedicated date handler.

By providing date handlers, several problems are resolved.

Dates entered by users of that language in almost any form can be parsed by Gramps.

The displayed dates will look clear and correct to the users.

Translators do not have to worry about translating regular expressions like (from|before|between).

Unfortunately, it turned out that the maintenance burden of date handlers across all the languages is very high compared to the regular maintenance overhead of strings in the translation database. Under 6926 we attempted to shift the balance back from coding to translation database. The new system also allows to support inflections for month names in the languages where it is needed. Date handler code still exists, but is much simpler and easier to maintain. See also the related open issues.

How to write a date handler

Since new code for Gramps is developed on trunk, the present instructions only refer to the current API. If you are interested in older API documentation, see the revision history on the wiki and in the source control.

Please use the generated documentation on the date handler framework as your programming reference. Here are the some highlights on how a date handler plugin for a new language should be developed.

The plugin must translate some strings in the translation database. Special conventions allow for finer linguistic control for inflected forms of month names for languages where this is needed.

The plugin must define two classes: one for parsing and one for displaying dates. Various degrees of customizations exist, depending on how different the standards for formatting the dates in your locale are from English.

The plugin should be registered in the system.

Before submitting it to the code base, it should be tested using the existing testing framework.

Localizing the date strings

Use the regular translation routine to translate the strings such as month names, date modifiers etc. These strings are stored in the class DateStrings, which is used both from the date parser and the date displayer.

Providing a localized date displayer class

If you inherit the default implementation, then all the dates will be displayed in iso style, without any visible difference between span and range dates.

The next level of customization is to select another pre-canned alternative, provided in the base class, as follows (like DateDisplayEN does):

display = DateDisplay.display_formatted

This will be sufficient for over half of the currently supported languages, that don't tweak the list of formats available for date display.

The core of display_formatted displays both regular and compound dates, and adorns them, if necessary, with modifiers (as in "before..."), quality indicators (as in "calculated..."), and calendar/non-standard year start information.

If your language requires inflections of month names dependent on modifiers, or, for instance, whether the date is a start or a stop date in a range, there are special strings you need to translate to indicate the appropriate inflection. Documentation for this advanced scenario can be found in the source code comments to DateDisplay.

If your language overrides Display.formats to something else, it should also override _display_calendar (or, separately, the _display_gregorian/_display_hebrew/... whatever methods that delegate to _display_calendar by default with the appropriate month lists). The override of _display_calendar should format the date according to the currently selected custom format, using self.format number as an index into the self.formats list.

Providing a localized date parser class

The parser class must provide parse() method. In fact, since the base class already defines such method, it is most likely that you will only need to re-define class constants and, maybe, the init_strings() method.

Localizing the date formats

If you use DateDisplay._display_calendar in your date displayer, whether directly or via DateDisplay.display_formatted, then your dates will be formatted using the translated format strings therein. The month names (whether short or long) in these formats are used to format an instance of the class Lexeme, initialized to the set of inflection forms from the translated month string.

These formats all have a warning in the translation database sending you to the entrance point to these docs. You have the following options in translating such a format string:

1) If you don't translate the string, it will just use the same order as in English, without any inflection selection.

2) If you omit the inflection selection in a format,
then the default inflection will be used, or, if your language has no inflections, just the one and only name for the month (see the documentation for class Lexeme for details). REMEMBER the format specifiers themselves must not be translated, otherwise there will be a runtime error in the formatting code. E.g.:

3) However, if your language does support inflections, here you should select the appropriate inflection for the month name,
using the same name of the inflection as you entered in the translated month string. Again, see the documentation for class Lexeme for details. E.g.:

Some of the strings need not be translated at all for your language if you don't use the inflection control feature of _display_calendar, e.g.:

#. first date in a span
#. You only need to translate this string if you translate one of the
#. inflect=_("...") with "from"
#: ../gramps/gen/datehandler/_datedisplay.py:160
msgid "from|{long_month} {year}"
msgstr ""

Documentation for this advanced scenario can be found in the source code comments to DateDisplay.

First, before changing your code, run the helper code to generate a snippet you can merge into your language's xx.po. See the documentation.

If you stop here, the handler should work already (strings previously in _grampslocale.py are now picked up from the translation database, and the rest remains the same). However, it'll likely have lots of redundant code and strings in it...

Then remove from your DateDisplayXX class any overrides of date strings from base class, such as
long_months, short_months, calendar, _mod_str, _qual_str -- these are now localized via the translation database.

NOTE: Leave _bce_str for now (it will be removed when 7064 is resolved).

In your DateParserXX class, remove the various ..._to_int that are now initialized based on the strings in DateStrings, i.e.:

Here we unite the alternative spellings Nisan and Nissan into the same lexeme.

Alternatively, you can extend the base string table that is initialized from the date strings explicitly from the code, as before. For example, here's how the Russian parser adds an alternative name for the Persian calendar:

If you don't have custom language-specific formats

If you override formats and it corresponds exactly to the formats in the DateDisplay.formats, only changing the strings inside to translate them to your language, then remove the formats override and any overrides of _display_gregorian/_display_julian/... and display methods.

Just alias your display to DateDisplay.display_formatted.

If you have custom language-specific formats

Consider rewriting your existing overrides of _display_gregorian and display to a single override of _display_calendar.
At the very least, you'll need to update the signature of your _display_* methods, to match _display_calendar. You may just add (and ignore) a trailing **kwargs if you're lazy...

Converted handlers

Add yours here when done!!!

FR (romjerome)

HR (josip)

RU (vassilii)

Registering the localized date handler

The module must register itself as the date handler. This is done by inserting the following code at the end of your _Date_xx.py file:

where MyDateParser and MyDateDisplay are the classes defined for parsing and displaying, and the items in quotes are language identifiers that may possibly be associated with your language. For example, different systems use ru, RU, ru_RU, koi8r, ru_koi8r, russian, Russian, ru_RU.koi8r, etc. to identify the Russian language.

Then you need to import the parser to Gramps by adding

import _Date_xx

line to __init__.py file located in the same directory.

That's it for the requirements. The example date handling plugins can be found in _Date_xx.py under the gramps/gen/datehandler directory.

Compare source and target dates in the entries on Events View. Alternatively, compare the birth and death dates in the People View. In the latter view, the failed tests are marked with a "Fail" tag on the person, you can use the "Customize view" option to add the "Tags" column to sort on the pass/fail status.