Researchers Study How the Brain Combines Visual Speech and Sound
Researchers are investigating how the brain combines visible speech cues with sound, particularly for people who use cochlear implants. The aim is to understand multisensory speech processing and, eventually, improve hearing technology.
In a crowded, noisy room, seeing a speaker's lips and gestures can help a listener make sense of unclear sounds. The physiological mechanisms behind that benefit remain incompletely understood. Edmund Lalor, an associate professor of biomedical engineering and neuroscience at the University of Rochester, noted that the visual cortex sits toward the back of the brain while auditory cortex is in the temporal lobe; how their information is integrated remains an open question.
Scientists have used noninvasive electroencephalography (EEG) to study responses to simple sounds such as beeps, clicks, and syllables. Lalor's group has investigated how the shapes made by the lips and tongue against the teeth can distinguish sounds such as “F” from “S” and “P” from “D” when the auditory signal is ambiguous.
Lalor now plans to study more natural, continuous audiovisual speech. The US National Institutes of Health is providing approximately $2.3 million over five years. The work follows an earlier NIH R01 grant and began with seed funding from the University of Rochester's Del Monte Institute for Neuroscience.
Studying cochlear implant users
The team plans to recruit 250 cochlear implant users, a group whose listening can be especially challenged by noise. Participants will wear EEG caps while watching and listening to audiovisual speech so researchers can measure their brain responses.
Lalor proposed that a person implanted at age one might develop auditory processing more like that of a normally hearing person, despite a year without auditory input. Someone implanted later, for example at age 12, might have missed a critical developmental period and rely differently—or more heavily—on visual information from a speaker's face. These are hypotheses the project will examine, not findings about individual implant users.
The project is being conducted with Matthew Dye, professor and director of the doctoral program in cognitive science and the National Center for Deaf Sensory, Perceptual, and Cognitive Ecology at Rochester Institute of Technology. Dye is also an adjunct faculty member at the University of Rochester Medical Center.
EEG records scalp electrical activity from many sources at once. Cochlear implants add their own electrical activity, making the signals harder to interpret. Lalor said the team will need extensive engineering work, including signal processing, engineering analysis, and computational modeling, to draw the most information from those recordings.
Ultimately, the researchers hope that a better understanding of audiovisual processing will inform technologies that improve speech understanding for deaf and hard-of-hearing people. No such improvement has yet been demonstrated by this planned project. Image credit in the original article: online image, with no specific creator identified. Original Hearing Review report.
