Multimodality, cognition and evaluation

The aim of this subproject
is to gain insights in how users interpret and evaluate multimodal
output, and whether or not users prefer particular combinations of
modalities to other combinations or prefer unimodal variants. This
knowledge can be used to make more informed
decisions for automatic multimedia generation. So far, we have carried
out several studies to investigate
multimodal output and modality effect, focusing on the question how
people process and evaluate unimodal and multimodal information in the
medical domain.

Van Hooijdonk and Krahmer (to appear) report on experiments that have
been
conducted to investigate which combinations of modalities (e.g., text
vs. picture vs. film clip; text and picture vs. text and film clip;
speech and picture vs. speech and film clip) are most efficient and
effective for illustrating RSI exercises, paying special attention to
cognitive load of learners and the special characteristics of
procedural learning (i.e., acquiring skills rather than knowledge).

Another
study focused on text-structure and spatial cognition. For this study,
we conducted an exploratory thinking aloud experiment in which users
had to find information in a large
medical web site. The thinking aloud protocols that were collected in
this study were analysed in order to gain insights in users’ spatial
orientation and search strategies in this domain. In particular, we
investigated the way in which spatial descriptions are used to
conceptualize users' search tasks. See Van Hooijdonk et al. (2004,
2005, 2006).

Recently
we have carried out a study investigating which (combinations of)
modalities people choose to answer different types of questions when
simulating a QA system. To this end we conducted two
experiments,
one in which we asked people to create multimodal answer presentations
for different medical questions, and one in which these multimodal
presentations were evaluated by other people (van Hooijdonk et al.,
2007).

For an overview of all results in this subproject, see Van Hooijdonk
(2008).