QA4MRE 2012 evaluation ongoing

The main objective of this exercise is to develop a methodology for evaluating Machine Reading systems through Question Answering and Reading Comprehension Tests. Systems should be able to extract knowledge from large volumes of text and use this knowledge to answer questions.

MAIN TASK

The Machine Reading task addresses the problem of building a bridge between knowledge encoded as natural text and the formal reasoning systems that need such knowledge. In contrast to text mining (or text harvesting, sometimes also called macro-reading), where the system reads and combines evidence from hundreds or thousands of texts, MR is the task of obtaining an in-depth understanding of just one, or a small number, of texts. In fact, the task will focus on the reading of single documents, where correct answers require some inference and the consideration of previously acquired background knowledge.

PILOT TASK

Beside the Main Task, also two pilot tasks are offered this year at QA4MRE:

Processing Modality and Negation for Machine Reading, aimed at evaluating whether systems are able to understand extra-propositional aspects of meaning like modality and negation. Modality is a grammatical category that allows expressing aspects related to the attitude of the speaker towards his/her statements. Negation is a grammatical category that allows changing the truth value of a proposition.

Machine Reading of Biomedical Texts about Alzheimer, aimed at setting questions in the Biomedical domain with a special focus on one disease, namely Alzheimer. This pilot task will explore the ability of a system to answer questions using scientific language. Texts will be taken from Medline (Medical Literature Analysis and Retrieval System Online) abstracts.