Data_Sheet_1_Competition between audiovisual correspondences aids understanding of interactions between auditory and visual perception.PDF

An audiovisual correspondence (AVC) refers to an observer's seemingly arbitrary yet consistent matching of sensory features across the two modalities; for example, between auditory pitch height and visual height or visual size. Research on AVCs frequently uses a speeded classification procedure in which participants are asked to rapidly classify the pitch of a sound accompanied either by a congruent or an incongruent visual object (e.g., high pitches are congruent with higher/smaller visual objects and incongruent with lower/larger visual objects). To investigate the strength of these pitch AVCs (height, size, spatial frequency, brightness, sharpness), trials where the height AVC competed with each other AVC in terms of pitch congruency were created. For example, when classifying pitch height, participants were presented with trials where both visual height and size were congruent or incongruent with pitch; additionally, there were trials where height was congruent but size was incongruent (i.e., high pitch matched with large object at high height) and trials where size was congruent but height was incongruent (i.e., high pitch matched with small object at low height). Based on previous work, congruency between pitch and height was expected to be more important than congruency between pitch and spatial frequency, brightness, sharpness, or size. As predicted, in all four studies, RTs when only height was congruent were just as fast as when both dimensions were congruent. In contrast, RTs when only spatial frequency, brightness, sharpness, or size was congruent (and height was incongruent) were just as slow as when both dimensions were incongruent. These results reinforce the superiority of the pitch-height AVC and can be interpreted based on the metaphor used for pitch in English, showing the importance of semantic/linguistic effects to understanding AVCs.视听对应（audiovisual correspondence, AVC）指观察者在两种感官模态间看似随机却具有一致性的感官特征匹配现象，例如听觉音高高度与视觉高度或视觉尺寸之间的匹配。针对视听对应现象的研究常采用快速分类范式，要求被试对伴随一致或不一致视觉客体的声音音高进行快速分类——例如，高音调与更高/更小的视觉客体一致，与更低/更大的视觉客体不一致。为探究这类音高视听对应（涵盖高度、尺寸、空间频率、亮度、锐度维度）的强度，研究者设计了在音高一致性上存在竞争的试次：即让高度维度的视听对应与其他维度的视听对应在音高一致性上产生冲突。例如，在完成音高高度分类任务时，向被试呈现以下几类试次：视觉高度与尺寸均与音高一致、均与音高不一致；视觉高度与音高一致但尺寸与音高不一致（即高音调匹配高度较高但尺寸较大的客体）；以及视觉尺寸与音高一致但高度与音高不一致（即高音调匹配高度较低但尺寸较小的客体）。基于既往研究基础，我们预期音高与视觉高度的一致性效应，相较于音高与空间频率、亮度、锐度或尺寸的一致性效应更为显著。实验结果与预期一致：在四项研究中，仅视觉高度与音高一致时的反应时，与双维度均一致时的反应时同样迅捷；与之相对，仅空间频率、亮度、锐度或尺寸与音高一致（而视觉高度与音高不一致）时的反应时，与双维度均不一致时的反应时同样缓慢。上述结果进一步印证了音高-高度视听对应的优势效应，且可结合英语中用于描述音高的隐喻得到解释，这凸显了语义与语言效应在理解视听对应现象中的重要性。