The power of sound: exploring the auditory influence on visual search efficiency

In a dynamic visual search environment, a synchronous and meaningless auditory signal (pip) that corresponds with a change in a visual target promotes the efficiency of visual search (pop out), which is known as the pip-and-pop effect. We conducted three experiments to investigate the mechanism of the pip-and-pop effect. Using the eye movement technique, we manipulated the interval rhythm (Exp. 1) and interval duration time (Exp. 2) of dynamic color changes in visual stimuli in the dynamic visual search paradigm to ensure that there was a significant pip-and-pop effect. In Exp. 3, we modulated the appearance of the sound by employing a visual-only condition, an auditory target condition (synchronized sounds), an auditory oddball condition (a high-frequency sound in a series of low-frequency sounds), an omitted oddball condition (an omitted sound in a series of sounds) and an auditory non-oddball condition (the last of the four sounds). We aim to clarify the role of audiovisual cross-modal information in the pip-and-pop effect by comparing different conditions. The search time results showed that a significant pip-and-pop effect was found for the auditory target, auditory oddball and auditory non-oddball conditions. The eye movement results revealed an increase in the fixation duration and a decrease in the number of fixations for the auditory target and auditory oddball conditions. Our findings suggest that the pip-and-pop effect is indeed a cross-modal effect. Furthermore, the interaction between auditory and visual information is necessary for the pip-and-pop effect, whereas auditory oddball stimuli attract attention and therefore moderate this effect. Our study provides a solution for the pip-and-pop effect mechanism in a dynamic visual search paradigm.在动态视觉搜索环境中，与视觉目标变化同步的无意义听觉信号（pip）可提升视觉搜索效率（即“凸显效应（pop out）”），该现象被称为pip-and-pop效应（pip-and-pop effect）。我们开展了三项实验以探究pip-and-pop效应的作用机制。借助眼动技术，我们在动态视觉搜索范式中对视觉刺激的动态颜色变化的间隔节律（实验1）与间隔时长（实验2）进行操纵，以确保能够观测到显著的pip-and-pop效应。实验3中，我们通过采用仅视觉条件、听觉目标条件（同步声音）、听觉偏差刺激（oddball）条件（低频声音序列中的高频声音）、省略偏差刺激条件（声音序列中缺失的声音）以及非偏差听觉刺激条件（四声序列中的最后一声）这几种实验范式，来调控声音的呈现。本研究旨在通过对比不同实验条件，阐明视听跨模态信息在pip-and-pop效应中的作用。搜索时长结果显示，在听觉目标、听觉偏差刺激及非偏差听觉刺激条件下，均观测到了显著的pip-and-pop效应。眼动结果表明，在听觉目标与听觉偏差刺激条件下，被试的注视时长显著增加，而注视次数显著减少。本研究结果表明，pip-and-pop效应本质属于跨模态效应。进一步而言，视听信息间的交互作用是pip-and-pop效应产生的必要条件，而听觉偏差刺激可通过吸引注意对该效应产生调节作用。本研究为动态视觉搜索范式下pip-and-pop效应的作用机制提供了阐释思路。