The TEMAA project is an LRE project carried out by researchers and
developers at Center for Sprogteknologi (Copenhagen), ISSCO
(Geneva), Stichting Taaltechnologie (Utrecht), Language Technology Group
(Edinburgh) and Claris Ireland (Dublin)
(see also (Thompson94)).

The users involved in this project are users of spelling checkers in
English, Italian, Danish and Dutch -- mother tongue users, as well as
foreign language users. The classes of users, apart from the managers
and end-users, which will profit from the results are: developers,
vendors and funders.

The objective of the project is to provide a generally applicable
framework for the evaluation of NLP products. The project also studies
project evaluation to some extent, but the focus is on adequacy
evaluation. The framework is expressed as a number of guidelines to be
followed when setting up an evaluation of a specific type of product.
These guidelines are implemented in an object-oriented environment to
facilitate modelling of user requirements and system characteristics.

The result is the development of a
Parameterized Test Bed (PTB).
Test materials for testing the lexical coverage of spelling checkers have
also been constructed. The TEMAA framework is modelled in CLIPS
following an object-oriented programming (OOP) paradigm. The project
has also developed an experimental tool supporting automatic spelling
checker checking (ASCC) which is interfaced with CLIPS. The tool is a
collection of programs in WINBATCH, C and Perl. The results of the
project will be documented in reports which will hopefully be widely
accepted as constituting a methodological framework for evaluation to
be further elaborated. Other results include an evaluation package,
covering some of the test materials necessary for the evaluation of
spelling checkers for Danish, English and Italian, a program, ASCC,
which automatically transforms correctly spelled words into misspelled
words and measures the ability of the spelling checker to find the
errors and to produce a good suggestion, and a prototype of a test
modelling tool for the description of an evaluation of some NLP
product, i.e. an assessment of the way in which system characteristics
are mapped onto different sets of user requirements. The final report
will give a good description of the system. The PTB implemented in
CLIPS is an experimental prototype aiming at demonstrating the
soundness of the general approach, covering a subset of the user
requirements and system parameters relevant to spelling and grammar
checkers. Likewise, only an interesting subset of typing and spelling
errors has been implemented in ASCC. The project envisages that the
results could be taken over by organisations such as ISO. As for new
partner types, the project envisages developers of the product type in
question and standardisation bodies or user organisations.