the principle of inverse effectiveness in audiovisual speech perception

We assessed how synchronous speech listening and lipreading affects speech recognition in acoustic noise. In simple audiovisual
perceptual tasks, inverse effectiveness is often observed, which holds that the weaker the unimodal stimuli, or the poorer their
signal-to-noise ratio, the stronger the audiovisual benefit. So far, however, inverse effectiveness has not been demonstrated for
complex audiovisual speech stimuli. Here we assess whether this multisensory integration effect can also be observed for the
recognizability of spoken words.
To that end, we presented audiovisual sentences to 18 native-Dutch normal-hearing participants, who had to identify the spoken
words from a finite list. Speech-recognition performance was determined for auditory-only, visual-only (lipreading) and auditoryvisual
conditions. To modulate acoustic task difficulty, we systematically varied the auditory signal-to-noise ratio. In line with a
commonly-observed multisensory enhancement on speech recognition, audiovisual words were more easily recognized than
auditory-only words (recognition thresholds of -15 dB and -12 dB, respectively).
We here show that the difficulty of recognizing a particular word, either acoustically or visually, determines the occurrence of
inverse effectiveness in audiovisual word integration. Thus, words that are better heard or recognized through lipreading,
benefit less from bimodal presentation.
Audiovisual performance at the lowest acoustic signal-to-noise ratios (45%) fell below the visual recognition rates (60%), reflecting
an actual deterioration of lipreading in the presence of excessive acoustic noise. This suggests that the brain may adopt a
strategy in which attention has to be divided between listening and lipreading.