Audio-visual Synchrony
Published on 3 Feb 2007 at 1:11 am.
1 Comment.
Filed under Neuroscience, Research.
Today our Physics Department had the pleasure of a visit by Lucas Parra who spoke to us about his recent work in Brain-Computer Interface. During our visit he told me something quite remarkable. I had known that the processing time of the human auditory system was actually different than the processing time of the human visual processing. This leads to some difficulties when one tries to imagine how the brain connects a visual stimulus to a sound stimulus. However, Lucas explained that the audio-visual asynchrony for speech is actually variable with the brain able to tolerate auditory delays up to a couple hundred milliseconds without the subject noticing the time difference.
I looked into this and found a paper by Grant, Wassenhove, and Poeppel titled “Discrimination of Auditory-Visual Synchrony“, which describes that time delays in the auditory stimulus with respect to the visual stimulus of -40 milliseconds to 240 milliseconds were not noticed by the subject. Apparently a visual speech stimulus (lips moving) and its associated auditory speech stimulus do not have to be so precisely timed for a person to perceive them as synchronous.
When you think about this, you realize that our perceptual processing must be this way. Light travels about one million times faster than sound. More specifically, light travels about 1 foot per nanosecond, whereas sound travels about a foot a millisecond. If a person is 20 feet away from you, the sound is delayed by 20 milliseconds with respect to the visual stimulus. This effect is more pronounced when the subject is 100 feet away, which leads to a 100 millisecond delay. To accomodate the slow speed of sound with respect to the speed of light, and the fact that sound sources can be located at varying distances from the perceiver, the brain must be able to accomodate sound delays on the order of hundreds of milliseconds. This reasoning explains why the brain works this way. However. it now remains to explain how this dynamic processing is implemented.
Last, the time delay in the other direction (-40 milliseconds) can be understood by realizing that a speech syllable is on the order of 40 milliseconds in length. Since a syllable is a basic unit of speech sound, it would make sense that some processing time on the order of the duration of a syllable needs to be accomodated.
Kevin H Knuth
Albany NY
Anita on 3 Feb 2007 at 11:50 am: 1
I’m still waking on this Saturday morn, but this puts me in the mind of the training I had that summer I taught reading classes. We tried to help students get past the subvocalization impulse (a holdover of the sustained-silent reading phase of learning to read) and rely on their visual intake text. I have no idea if the science was sound, but it worked.
Back to the coffee and the bad Val Kilmer movie on the telly!