Data_Sheet_3_Understanding Events by Eye and Ear: Agent and Verb Drive Non-anticipatory Eye Movements in Dynamic Scenes.pdf

As Macnamara (1978) once asked, how can we talk about what we see? We report on a study manipulating realistic dynamic scenes and sentences aiming to understand the interaction between linguistic and visual representations in real-world situations. Specifically, we monitored participants’ eye movements as they watched video clips of everyday scenes while listening to sentences describing these scenes. We manipulated two main variables. The first was the semantic class of the verb in the sentence and the second was the action/motion of the agent in the unfolding event. The sentences employed two verb classes–causatives (e.g., break) and perception/psychological (e.g., notice)–which impose different constraints on the nouns that serve as their grammatical complements. The scenes depicted events in which agents either moved toward a target object (always the referent of the verb-complement noun), away from it, or remained neutral performing a given activity (such as cooking). Scenes and sentences were synchronized such that the verb onset corresponded to the first video frame of the agent motion toward or away from the object. Results show effects of agent motion but weak verb-semantic restrictions: causatives draw more attention to potential referents of their grammatical complements than perception verbs only when the agent moves toward the target object. Crucially, we found no anticipatory verb-driven eye movements toward the target object, contrary to studies using non-naturalistic and static scenes. We propose a model in which linguistic and visual computations in real-world situations occur largely independent of each other during the early moments of perceptual input, but rapidly interact at a central, conceptual system using a common, propositional code. Implications for language use in real world contexts are discussed.

正如麦克纳马拉（Macnamara）1978年曾提出的疑问：我们该如何谈论所目睹的事物？本研究通过操控真实动态场景与语句，旨在探究现实场景中语言表征与视觉表征间的交互机制。具体而言，本研究监测了被试在观看日常场景视频片段并聆听描述对应场景的语句时的眼动轨迹。本研究操控了两项核心变量：其一为语句中动词的语义类别，其二为事件展开过程中施动者的动作/运动状态。实验所用语句包含两类动词——使役动词（如break，打破）与感知/心理动词（如notice，注意到），二者对充当语法补语的名词施加了不同的语义约束。实验场景所呈现的事件中，施动者可分为三种状态：朝向目标物体（始终为动词补语名词的所指对象）移动、远离目标物体，或保持中立状态执行特定活动（如烹饪）。场景与语句实现了同步：动词起始时刻与施动者朝向或远离目标物体的首个视频帧完全对应。研究结果显示，施动者运动状态存在显著效应，但动词语义约束的效应较弱：仅当施动者朝向目标物体移动时，使役动词相较于感知动词，会让被试更多地关注其语法补语的潜在所指对象。至关重要的是，本研究未发现由动词驱动的朝向目标物体的预期性眼动，这与采用非自然静态场景的相关研究结论相悖。本研究提出一种模型：现实场景中的语言与视觉计算在感知输入的早期阶段基本相互独立，但随后会通过共享的命题编码在中央概念系统中快速实现交互。本文最后讨论了该研究对现实场景中语言使用的启示。