The visual speech head start improves perception and reduces superior temporal cortex responses to auditory speech

Visual information about speech content from the talker's mouth is often available before auditory information from the talker's voice. Previously we demonstrated that audiovisual speech selectively enhances activity in regions of the early visual cortex representing the mouth of the talker (Ozker et al., 2018b). Here we examined perceptual and neural responses to words with and without this visual head start. For both types of words, perception was enhanced by viewing the talker's face, but the enhancement was significantly greater for words with a head start. Neural responses were measured from electrodes implanted over auditory association cortex in the posterior superior temporal gyrus (pSTG) of epileptic patients. The presence of visual speech suppressed responses to auditory speech, more so for words with a visual head start. We suggest that the head start inhibits representations of incompatible auditory phonemes, increasing perceptual accuracy and decreasing total neural respons...

说话者唇部传递的视觉言语信息，往往早于其嗓音带来的听觉言语信息被接收。此前我们的研究已证实，视听言语（audiovisual speech）可选择性增强早期视觉皮层（early visual cortex）中负责映射说话者唇部区域的神经活动（Ozker等人，2018b）。本研究针对具备该视觉前置优势与不具备该优势的词汇，分别考察了其知觉与神经响应。两类词汇的知觉感知均能通过观看说话者面部得到提升，但具备视觉前置优势的词汇，其感知提升幅度显著更高。我们通过植入癫痫患者后颞上回（posterior superior temporal gyrus, pSTG）听觉联合皮层表面的电极，记录了神经响应。视觉言语的存在会抑制听觉言语的神经响应，对于具备视觉前置优势的词汇，这种抑制效应更为显著。我们推测，该视觉前置优势会抑制不匹配听觉音素的神经表征，从而提升知觉准确率并降低总神经响应……