CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

Humans are able to perceive, understand and reason about physical events. Developing models with similar physical understanding capabilities is a long-standing goal of artificial intelligence. As a step towards this goal, in this work, we introduce CRAFT, a new visual question answering dataset that requires causal reasoning about physical forces and object interactions. It contains 58K video and question pairs that are generated from 10K videos from 20 different virtual environments, containing various objects in motion that interact with each other and the scene. Two question categories from CRAFT include previously studied <em>descriptive</em> and <em>counterfactual</em> questions. Besides, inspired by the theories of force dynamics in cognitive linguistics, we introduce new question categories that involve understanding the interactions of objects through the notions of <em>cause</em>, <em>enable</em>, and <em>prevent</em>. Our results demonstrate that even though these tasks seem to be simple and intuitive for humans, the evaluated baseline models, including existing state-of-the-art methods, do not yet deal with the challenges posed in our benchmark dataset.

人类能够对物理事件进行感知、理解与推理。研发具备类似物理认知能力的模型，始终是人工智能领域的长期目标。为推进这一目标，本研究推出了CRAFT数据集——一款全新的、要求针对物理作用力与物体交互开展因果推理的视觉问答（Visual Question Answering）数据集。该数据集包含5.8万组视频与问答对，其数据来源于20个不同虚拟环境中的1万段视频，场景内设有各类处于运动状态且彼此间、与场景环境存在交互的物体。CRAFT数据集包含两类此前已被广泛研究的题型：<em>描述性</em>问题与<em>反事实</em>问题。此外，受认知语言学中的力动态理论启发，本研究还新增了三类基于<em>引发</em>、<em>使能</em>与<em>阻止</em>概念来理解物体交互关系的题型。我们的实验结果表明，尽管这些任务对人类而言简单直观，但经测试的基线模型（包括现有顶尖方法）仍无法有效应对该基准数据集所提出的挑战。