Table_3_Cross-Modal Interaction Between Auditory and Visual Input Impacts Memory Retrieval.docx

How we perceive and learn about our environment is influenced by our prior experiences and existing representations of the world. Top-down cognitive processes, such as attention and expectations, can alter how we process sensory stimuli, both within a modality (e.g., effects of auditory experience on auditory perception), as well as across modalities (e.g., effects of visual feedback on sound localization). Here, we demonstrate that experience with different types of auditory input (spoken words vs. environmental sounds) modulates how humans remember concurrently-presented visual objects. Participants viewed a series of line drawings (e.g., picture of a cat) displayed in one of four quadrants while listening to a word or sound that was congruent (e.g., “cat” or ), incongruent (e.g., “motorcycle” or ), or neutral (e.g., a meaningless pseudoword or a tonal beep) relative to the picture. Following the encoding phase, participants were presented with the original drawings plus new drawings and asked to indicate whether each one was “old” or “new.” If a drawing was designated as “old,” participants then reported where it had been displayed. We find that words and sounds both elicit more accurate memory for what objects were previously seen, but only congruent environmental sounds enhance memory for where objects were positioned – this, despite the fact that the auditory stimuli were not meaningful spatial cues of the objects’ locations on the screen. Given that during real-world listening conditions, environmental sounds, but not words, reliably originate from the location of their referents, listening to sounds may attune the visual dorsal pathway to facilitate attention and memory for objects’ locations. We propose that audio-visual associations in the environment and in our previous experience jointly contribute to visual memory, strengthening visual memory through exposure to auditory input.

人类对周遭环境的感知与认知，受既往经验与既有世界表征的共同影响。自上而下的认知加工过程，例如注意与预期，能够改变我们对感官刺激的处理方式，既存在于单模态内（例如听觉经验对听觉感知的影响），也存在于跨模态间（例如视觉反馈对声音定位的影响）。本研究证实，不同类型听觉输入（口语词汇与环境声音）的接触经验，会调节人类对同时呈现视觉物体的记忆能力。实验中，被试依次观看呈现在四个象限之一的系列线条画（例如猫咪图像），同时聆听与该图像匹配（例如"猫咪"或对应环境音）、不匹配（例如"摩托车"或对应环境音）或中性（例如无意义伪词或音调蜂鸣声）的词汇或声音。在编码阶段结束后，向被试呈现原有线条画与全新线条画，并要求其判断每张图像为"旧"（此前见过）或"新"（首次出现）。若判定为"旧"，则要求被试报告该图像此前呈现的象限位置。研究结果显示，词汇与环境声音均能提升被试对此前所见物体类别的记忆准确性，但仅一致匹配的环境声音能够增强被试对物体呈现位置的记忆——尽管听觉刺激本身并未提供物体在屏幕上位置的有效空间线索。鉴于真实世界聆听场景中，环境声音（而非词汇）通常精准地源自其所指代物体的位置，因此聆听环境声音或许能够调节视觉背侧通路（visual dorsal pathway），以促进对物体位置的注意与记忆。本研究提出，环境与既往经验中的视听关联会共同作用于视觉记忆，并通过接触听觉输入强化视觉记忆。