audio-visual integration

Although we rarely stop to appreciate its complexity, our everyday experience of the world is dependent upon the seamless integration of information in different modalities. The process by which this “illusion of unity” arises from a constant stream of sights and sounds is both complex and informative. Due to its multi-sensory nature, music provides an especially rich and vibrant domain in which to explore audio-visual integration, particularly through an illusion in which percussionists use body movements to shape the way performances “sound.”

The differences in how we perceive and understand the motion of an inanimate object versus a biological being.

What makes biological motion unique from non-biological? How far can reduce the qualities of human movement until it no longer appears human?

Although much of our work on this topic is inspired by musical situations, this research holds broad implications for theories of sensory integration that generalize beyond music listening. For example, further exploration of this illusion builds a strong case for the role of causality as a cue for audio-visual integration. Furthermore, conflicts between the patterns of audio-visual integration observed in our paradigm and previous research on this topic have now led to new insights on the role of amplitude envelope in auditory perception general, as well as its role in audio-visual integration in particular.

We are currently working with a network of collaborators interested in applying our techniques and approaches to explore audio visual integration in individuals with autism spectrum disorder, as well as integration in hearing impaired populations. Both of these studies will help extend our basic research by informing treatments for conditions that affect many individuals around the world. For more information on this work, please see an archived PDF of Nedra Floyd-Pautler’s feature story on our work on the Hearing Lab.