Conveying Movement in Music and Prosody

We investigated whether acoustic variation of musical properties can analogically convey descriptive information about an object. Specifically, we tested whether information from the temporal structure in music interacts with perception of a visual image to form an analog perceptual representation as a natural part of music perception. In Experiment 1, listeners heard music with an accelerating or decelerating temporal pattern, and then saw a picture of a still or moving object and decided whether it was animate or inanimate – a task unrelated to the patterning of the music. Object classification was faster when musical motion matched visually depicted motion. In Experiment 2, participants heard spoken sentences that were accompanied by accelerating or decelerating music, and then were presented with a picture of a still or moving object. When motion information in the music matched motion information in the picture, participants were similarly faster to respond. Fast and slow temporal patterns without acceleration and deceleration, however, did not make participants faster when they saw a picture depicting congruent motion information (Experiment 3), suggesting that understanding temporal structure information in music may depend on specific metaphors about motion in music. Taken together, these results suggest that visuo-spatial referential information can be analogically conveyed and represented by music and can be integrated with speech or influence the understanding of speech.

本研究旨在探究音乐属性的声学变化能否类比地传递关于某一客体的描述性信息。具体而言，我们检验了音乐的时间结构信息是否会与视觉图像知觉产生交互，从而形成类比性知觉表征，且该过程作为音乐知觉的自然组成部分存在。在实验1中，被试聆听具有加速或减速时间模式的音乐，随后观看静止或运动客体的图片，并判断该客体是有生命还是无生命——该任务与音乐的模式无关。当音乐的运动模式与视觉呈现的运动模式一致时，客体分类的反应速度更快。在实验2中，被试聆听伴随加速或减速音乐的口语句子，随后观看静止或运动客体的图片。当音乐中的运动信息与图片中的运动信息一致时，被试的反应速度同样更快。然而，在实验3中，仅具备快慢变化但无加速减速的时间模式，并未让被试在观看呈现一致运动信息的图片时反应更快。这表明，对音乐时间结构信息的理解可能依赖于关于音乐运动的特定隐喻。综合来看，上述结果表明，视觉空间指称信息可通过音乐进行类比性传递与表征，且能够与言语进行整合，或影响对言语的理解。