Electrophysiological Indices of Audiovisual Speech Perception: Beyond the McGurk Effect and Speech in Noise. Supplementary Material

Visual information on a talker’s face can influence
what a listener hears. Commonly used approaches to study this include
mismatched audiovisual stimuli (e.g., McGurk type stimuli) or visual speech in
auditory noise. In this paper we discuss potential limitations of these
approaches and introduce a novel visual phonemic restoration method. This
method always presents the same visual stimulus (e.g., /ba/) dubbed with a
matched auditory stimulus (/ba/) or one that has weakened consonantal information
and sounds more /a/-like). When this reduced auditory stimulus (or /a/)
is dubbed with the visual /ba/, a visual influence will result in effectively
‘restoring’ the weakened auditory cues so that the stimulus is perceived as a
/ba/. An oddball design in which participants are asked to detect the /a/
among a stream of more frequently occurring /ba/s while either a speaking face
or face with no visual speech was used. In addition, the same paradigm was
presented for a second contrast in which participants detected /pa/ among
/ba/s, a contrast which should be unaltered by the presence of visual speech.
Behavioral and some ERP findings reflect the expected phonemic restoration for
the /ba/ vs. /a/ contrast; specifically, we observed reduced accuracy
and P300 response in the presence of visual speech. Further, we report an
unexpected finding of reduced accuracy and P300 response for both speech
contrasts in the presence of visual speech, suggesting overall modulation of
the auditory signal in the presence of visual speech. Consistent with this, we
observed a mismatch negativity (MMN) effect for the /ba/ vs. /pa/ contrast only
that was larger in absence of visual speech.We discuss the potential utility
for this paradigm for listeners who cannot respond actively, such as infants
and individuals with developmental disabilities.