Abstract

Recent empirical evidence suggests that language-mediated eye gaze
is partly determined by level of formal literacy training. Huettig, Singh and
Mishra (2011) showed that high-literate individuals' eye gaze was closely time
locked to phonological overlap between a spoken target word and items presented
in a visual display. In contrast, low-literate individuals' eye gaze was not
related to phonological overlap, but was instead strongly influenced by semantic
relationships between items. Our present study tests the hypothesis that this
behavior is an emergent property of an increased ability to extract phonological
structure from the speech signal, as in the case of high-literates, with
low-literates more reliant on more coarse grained structure. This hypothesis was
tested using a neural network model, that integrates linguistic information
extracted from the speech signal with visual and semantic information within a
central resource. We demonstrate that contrasts in fixation behavior similar to
those observed between high and low literates emerge when models are trained on
speech signals of contrasting granularity.