Cross-Modal Interaction between Vision and Hearing: A Speed-Accuracy Analysis

Article excerpt

Cross-modal facilitation of response time (RT) is said to occur in a selective attention task when the introduction of an irrelevant sound increases the speed at which visual stimuli are detected and identified. To investigate the source of the facilitation in RT, we asked participants to rapidly identify the color of lights in the quiet and when accompanied by a pulse of noise. The resulting measures of accuracy and RT were used to derive speed-accuracy trade-off functions (SATFs) separately for the noise and the no-noise conditions. The two resulting SATFs have similar slopes and intercepts and, thus, can be treated as overlapping segments of a single function. That speeded identification of color with and without the presence of noise can be described by one SATF suggests, in turn, that cross-modal facilitation of RT represents a change in decision criterion induced by the auditory stimulus. Analogous changes in decision criteria might also underlie other measures of cross-modal interactions, such as auditory enhancement of brightness judgments.

Multisensory research is a rapidly growing field that considers the ways in which different senses interact (Calvert, Spence, & Stein, 2004). These interactions are deemed crucial to the elaborate process of building a coherent picture of the environment. For example, visual and auditory cues serve together to help pinpoint the source of incoming sounds. When these cues are dissociated, gross errors may occur in sound localization (the ventriloquist effect; Alais & Burr, 2004; Jack & Thurlow, 1973).

One of the basic phenomena that emerged from early multisensory research is cross-modal facilitation of response time (RT) in a selective attention task (Nickerson, 1973; Todd, 1912). When people are asked to respond to a visual stimulus-say, to determine the lateral position of a flash of light-their choice RT (CRT) is likely to be shorter if the visual stimulus is accompanied by an irrelevant auditory signal (Bernstein, Clark, & Edelstein, 1969; Bernstein & Edelstein, 1971; Simon & Craft, 1970). This result is surprising because the auditory signal, by definition, carries no information relevant to the visual choice. Nevertheless, the effect is robust and has been replicated under diverse experimental settings. What mechanism might account for the cross-modal facilitation of visual CRT?

Two general classes of mechanisms have been proposed to account for the facilitation, one class being sensory and the other decisional. Sensory mechanisms include prominently the hypothesis of energy summation (or integration): that some of the energy in the auditory stimulus combines with the energy in the visual stimulus at a relatively early stage of information processing, thereby increasing the effective visual intensity (Bernstein et al., 1969; Bernstein, Rose, & Ashe, 1970). In other words, the effect of the irrelevant acoustic signal is comparable, by this hypothesis, to that of increasing the luminance of the visual stimulus. In the language of signal detection theory, this account implies that cross-modal facilitation in CRT occurs because of a shift in visual sensitivity wrought by adding acoustic energy. Other sensory mechanisms are also possible. For example, the addition of an auditory stimulus could serve as a signal that reduces temporal uncertainty, thereby increasing the ratio of signal to noise in the visual system and, hence, visual sensitivity.

Alternatively, cross-modal facilitation of RT might reflect the operation of decisional processes. An example is the preparation enhancement hypothesis (Nickerson, 1973), which suggests that the effect of the auditory signal is essentially the same as that of a warning signal that prompts the observer to respond (see also Posner, Nissen, & Klein, 1976). Rather than increasing the detect-ability or discriminability of the visual target, however, this hypothesis states that the auditory stimulus causes the observers to rely on less information when making their decision and, thus, to respond sooner. …