Expectations about dynamic visual objects facilitates early sensory processing of congruent sounds

In everyday life, the perception of a moving object can lead to the expectation of an object’s sound, yet little is known about how visual expectations influence early auditory processing. We examined how dynamic visual input – an object moving continuously across the visual field – influences early auditory processing of a sound that is either congruent with the object’s motion, and thus likely perceived as being part of the visual object, or incongruent with the object’s motion. In Experiment 1, EEG activity was recorded from 29 adults who passively viewed a ball that appeared either on the far left or right boundary of a display and continuously traversed along the horizontal midline to make contact and elicit a bounce sound off the opposite boundary. Our main analysis focused on the auditory-evoked event-related potential. For audio-visual (AV) trials, a knocking sound accompanied the visual input the moment the ball made contact with the opposite boundary (AV-synchronous), or the sound occurred shortly before contact (AV-asynchronous). We also included audio-only and visual-only trials. For Experiment 1, AV-synchronous sounds elicited an earlier and attenuated auditory response relative to AV-asynchronous or audio-only events. Experiment 2 was conducted to examine the roles of expectancy and multisensory integration in influencing this early auditory response. In addition to the audio-only, AV-synchronous, and AV-asynchronous conditions, 19 adults were shown a ball that became visually occluded prior to reaching the boundary of the display, but elicited an expected knocking sound at the point of occluded collision. Here, the auditory response during the AV-occluded condition resembled the AV-synchronous condition, suggesting that expectations induced by a moving object can influence early auditory processing. Broadly, the results suggest that dynamic visual stimuli can help generate expectations about the timing of auditory events, which then facilitates the processing of auditory information that matches these expectations. Each version of this online data repository reflects revisions made to the manuscript post-peer review. The EEG/ERP data attached (.set/.fdt files) were processed in MATLAB using EEGLAB/ERPLAB software. These files can be found in the "Segmented ERP Data" folder. EEG/ERP data were processed using the scripts contained within the "EEG ERP Scripts (MATLAB)" folder. The EEG data were processed using the specifications outlined below. N1 and P2 peak amplitude and latency stats and other relevant data sets are contained within the "Data Sets (Long Format)" folder. All statistical analyses were conducted in R, and the scripts used to conduct such analyses can be found in the "R Scripts" folder. The E-prime scripts and the stimuli used for each experiment can be found in the "Experiment Stimuli and Scripts" folder. Raw EEG data sets are also available for download via a separate online data repository (osf.io/d245g/).日常生活中，对运动物体的感知会引发对该物体发声的预期，但目前学界对视觉预期如何影响早期听觉加工的认知仍较为有限。本研究探讨了动态视觉输入——即持续在视野中移动的物体——如何影响与该物体运动一致（即可能被感知为视觉物体的一部分）或不一致的声音的早期听觉加工。 实验1中，研究人员记录了29名成人的脑电图（Electroencephalogram, EEG）活动，这些被试被动观看一个小球：该小球会出现在屏幕的最左侧或最右侧边界，并沿水平中线持续移动，直至触碰对侧边界并引发弹跳声。本研究的主要分析聚焦于听觉诱发性事件相关电位（Event-Related Potential, ERP）。在视听（Audio-Visual, AV）试次中，当小球触碰对侧边界的瞬间，会伴随敲击声（视听同步条件），或声音在触碰前短暂出现（视听异步条件）。此外实验还设置了纯听觉与纯视觉试次。实验1结果显示，相较于视听异步或纯听觉事件，视听同步条件下的声音会引发更早且振幅衰减更显著的听觉反应。 实验2旨在验证预期与多感觉整合在影响该早期听觉反应中的作用。除纯听觉、视听同步与视听异步条件外，研究向19名成人展示了一个小球：该小球在抵达屏幕边界前会被视觉遮挡，但会在预期的碰撞点发出敲击声。在此条件下，视听遮挡组的听觉反应与视听同步组相似，这表明运动物体引发的预期能够影响早期听觉加工。 总体而言，本研究结果提示动态视觉刺激可帮助个体建立对听觉事件发生时机的预期，进而促进与这些预期相符的听觉信息的加工。 本在线数据仓库的各个版本均反映了稿件经同行评审后的修订内容。附带的脑电图/事件相关电位（EEG/ERP）数据（.set/.fdt格式文件）通过MATLAB软件结合EEGLAB/ERPLAB工具包进行处理，这些文件可在“分段ERP数据”文件夹中找到。脑电图/事件相关电位数据的处理使用了“EEG ERP Scripts (MATLAB)”文件夹中的脚本。脑电图数据的处理遵循下述规范。N1与P2峰的振幅、潜伏期统计数据及其他相关数据集收录于“数据集（长格式）”文件夹中。所有统计分析均在R语言环境中完成，相关分析脚本可在“R脚本”文件夹中获取。各实验使用的E-prime脚本与刺激材料可在“实验刺激与脚本”文件夹中找到。原始脑电图数据集可通过独立的在线数据仓库（osf.io/d245g/）下载。