Multisensory expectations about dynamic visual objects facilitates early sensory processing of congruent sounds

In everyday life, the perception of a moving object can lead to the expectation of an object’s sound, yet little is known about how visual expectations influence early auditory processing. We examined how dynamic visual input – an object moving continuously across the visual field – influences early auditory processing of a sound that is either congruent with the object’s motion, and thus likely perceived as being part of the visual object, or incongruent with the object’s motion. In Experiment 1, EEG activity was recorded from 29 adults who passively viewed a ball that appeared either on the far left or right boundary of a display and continuously traversed along the horizontal midline to make contact and elicit a bounce sound off the opposite boundary. Our main analysis focused on the N1 component of the auditory-evoked event-related potential. For audio-visual (AV) trials, a knocking sound accompanied the visual input the moment the ball made contact with the opposite boundary (AV-synchronous), or the sound occurred shortly before contact (AV-asynchronous). We also included audio-only and visual-only trials. For Experiment 1, AV-synchronous sounds elicited an earlier and attenuated auditory N1 response relative to AV-asynchronous or audio-only events. Experiment 2 was conducted to examine the roles of expectancy and multisensory integration in influencing this early auditory response. In addition to the audio-only, AV-synchronous, and AV-asynchronous conditions, 19 adults were shown a ball that became visually occluded prior to reaching the boundary of the display, but elicited an expected knocking sound at the point of occluded collision. Here, the auditory response during the AV-occluded condition resembled the AV-synchronous condition, suggesting that expectations induced by a moving object can influence early auditory processing. Broadly, the results suggest that dynamic visual stimuli can help generate expectations about the timing of auditory events, which then facilitates the processing of auditory information that matches these expectations. The second draft of this online data repository reflects revisions made to the manuscript post-peer review. The EEG/ERP data attached (.set/.fdt files) were processed in MATLAB using EEGLAB/ERPLAB software. These files can be found in the "Segmented ERP Data" folder. EEG/ERP data were processed using the scripts contained within the "EEG ERP Scripts (MATLAB)" folder. The EEG data were processed using the specifications outlined below. N1 and P2 peak amplitude and latency stats and other relevant data sets are contained within the "Data Sets (Long Format)" folder. Lastly, all statistical analyses were conducted in R, and the scripts used to conduct such analyses can be found in the "R Scripts" folder.

日常生活中，对运动物体的感知可引发对该物体发声的预期，但目前学界对视觉预期如何影响早期听觉加工的认知仍较为有限。本研究旨在探究动态视觉输入——即持续在视野中运动的物体——如何影响早期听觉加工，具体针对与物体运动一致（即可能被感知为视觉物体的一部分）或不一致的声音的早期听觉加工过程。实验1中，研究招募29名成年人，采用被动观看范式：被试观看一个先出现在显示屏最左侧或最右侧边界的小球，该小球沿水平中线持续移动，直至触碰显示屏对侧边界并引发弹跳声。本研究的核心分析聚焦于听觉诱发事件相关电位（event-related potential, ERP）的N1成分。在视听（audio-visual, AV）试次中，当小球触碰对侧边界的瞬间，同步伴随敲击声（AV同步条件），或在触碰前短时间呈现该声音（AV异步条件）。本实验同时设置了纯听觉试次与纯视觉试次。实验1结果显示，相较于AV异步条件或纯听觉试次，AV同步条件下的声音可诱发更早且幅值更低的听觉N1反应。为探究预期与多感觉整合在调控该早期听觉反应中的作用，本研究开展了实验2。除纯听觉、AV同步与AV异步条件外，实验2招募19名成年人，向其呈现一个在抵达显示屏边界前发生视觉遮蔽的小球，但在理论碰撞点呈现预期的敲击声。该AV遮蔽条件下的听觉反应模式与AV同步条件相似，表明运动物体引发的视觉预期可调控早期听觉加工过程。总体而言，本研究结果表明，动态视觉刺激可帮助个体建立对听觉事件发生时序的预期，进而促进与该预期匹配的听觉信息的加工。本在线数据仓库的第二版修订稿反映了同行评议后对稿件的修改内容。附件中的脑电图（electroencephalogram, EEG）/事件相关电位（ERP）数据（.set/.fdt格式文件）通过MATLAB软件结合EEGLAB与ERPLAB工具包完成预处理。此类文件可在「分段ERP数据」文件夹中获取。EEG/ERP数据的预处理采用「EEG ERP脚本（MATLAB）」文件夹内的脚本完成。脑电图数据的预处理遵循下述规范。N1与P2峰幅值、潜伏期统计数据及其他相关数据集均存放于「数据集（长格式）」文件夹中。最后，所有统计分析均通过R语言完成，相关分析脚本可在「R脚本」文件夹中找到。