Currently this page contains only preliminary information.
Please watch this space for updates.

Goals

The primary goal of the First Shared Task on Parsing Morphologically-Rich Languages was to bring forward work on parsing morphologically ambiguous input in both dependency and constituency parsing, and to show the state of the art for MRLs. In the longer term, we aim to provide streamlined data sets and evaluation metrics, thus improving the comparability of cross linguistic work on parsing MRLs. The shared task featured tracks in constituency parsing and in dependency parsing, in gold as well as in realistic scenarios (the realistic scenario has no gold tokenization, no gold part-of-speech tags and morphological features).

This year's edition will allow and favor the use of large unlabel data set. In order to correctly evaluate the improvment brought by the use of semi-supervized models, all annotated data and evaluation process will remain the same.

Annotated Data Set

The participants will be provided with data from 9 different languages (Arabic, Basque, French, German, Hebrew, Hungarian, Korean, Polish,Swedish). The data are available in Penn Treebank bracketing format, CoNLL-X format and optionally in TiGerXML.
In order to ease cross-linguistic comparisons, the data set will also be released within a common size setting (ie, treebanks of 5000 sentences).
All treebanks (dep. and const.) are aligned at the sentence, token and POS levels.

Metrics

Gold Tokens Scenarios:

We'll use two metrics: Parseval (Evalb, (Black et al, 91) and LeafAncestor (Sampson and Babarczy, 2003). With a modified version (from Sancl 2012 (Petrov and Mc Donald, 2012) that penalises unparsed trees for the former and with an implementation from Wagner (2012) for the latter.

LeafAncestor: parse_la.py (please read the disclaimer on top of the file)

Note: as oppposed to the common usage in the parsing communities, all constituency results are given for sentences of all lenght and all tokens are evaluated (including punctuation tokens). For both Evalb and LeafAncestor, the labels {TOP, S1, ROOT, VROOT} are stripped off.

Multi Word Expressions evaluation:

The French data set contains MWEs annotated at the morpho syntactic level. We're currently evaluating them for the dependency track only. (see wiki page )

Predicted Tokens Scenarios:

Dependency and Constituent Structures

We'll use TedEval (Tsarfaty et al 2010,2011,2012) in its realistic framework (namely a test file with its own mapping between predicted tokens and source tokens is evaluated upon a gold file and the gold token mapping). TedEval is available here: Tedeval 2.2.

We developped a set of wrappers that use MaltParser's reprojectiver (Nivre & Nilsson, 2005). Wrappers are available here: TedWrappers_20131015.tar.gz

Getting the Shared Task Data Set

All data but Arabic are freely available under the same conditions as during the shared task.
Unless stated otherwise by their original licenses, any commercial exploitation of treebank data,
derived parsing or tagging models are prohibited. Those data set are made available for
reproductibility's sake and in the hope that this shared task data will provide inspiration
for the design and evaluation of future parsing systems for these languages.

The Arabic data we provided is based on the LDC's ATB 4.1, 3.1 and 3.2, then converted to
both Columbia's CaTib Dependency Treebank (Habash & Roth, 2009) and to Stanford's preprocessed version
of the ATB (Green & Manning, 2010).
It is to be made available soon by the LDC via its usual channels. Contact us at spmrl.sharedtask@gmail.com
if you absolutely need the data urgently, we'll made available our (huge) set of scripts we developed
to create the data.

Acknowledgements

For their precious help preparing the SPMRL 2013 and 2014 Shared Task and for
allowing their data to be part of it, we warmly thank the Linguistic
Data Consortium, the Knowledge Center for Processing Hebrew (MILA),
the Ben Gurion University, Columbia University, Institute of Computer
Science (Polish Academy of Sciences), Korea Advanced Institute of
Science and Technology, University of the Basque Country,
Uppsala University, University of Stuttgart, University of
Szeged and University Paris Diderot (Paris 7).
We are also very grateful to the Philosophical Faculty of the Heinrich-Heine
Universität Düsseldorf for hosting the shared task data via their dokuwiki.