CRIPP-VQA
收藏资源简介：
视频通常会捕获对象，它们的运动以及不同对象之间的交互。尽管现实世界的物体具有与之相关联的物理属性，但其中许多属性 (例如质量和摩擦系数) 并未直接由成像管道捕获。但是，可以通过利用相对物体运动和碰撞引入的动力学的提示来估计这些属性。在本文中，我们介绍了一种新的视频问答任务，用于从视频中推理场景中对象的隐式物理属性。对于此任务，我们介绍了一个dataset-CRIPP-VQA，其中包含运动中的对象的视频，并带有有关操作效果 (例如删除，添加或替换对象) 的假设/反事实问题，有关计划的问题 (为达到特定目标而选择要执行的操作)，以及有关对象可见属性的描述性问题。我们在cripp-vqa的两个测试设置上对现有视频问答模型的性能进行基准测试: i.d.以及不分布设置，其中包含质量，摩擦系数和初始速度值的对象，这些对象在训练分布中看不到。在回答有关对象的隐式属性 (本文的重点) 和显式属性 (先前工作的重点) 的问题方面，我们的实验揭示了令人惊讶的显着性能差距。
Videos typically capture objects, their motions, and interactions between different entities. Although real-world objects possess physical properties associated with them, many of these attributes—such as mass and coefficient of friction—are not directly captured by the imaging pipeline. However, these properties can be estimated by leveraging dynamic cues introduced by relative object motions and collisions. In this paper, we propose a novel video question answering (VideoQA) task for reasoning about the implicit physical properties of objects in a scene from videos. For this task, we introduce CRIPP-VQA, a dataset consisting of videos of moving objects, paired with three types of questions: counterfactual/hypothetical questions about the effects of operations (e.g., deleting, adding, or replacing objects), planning questions that involve selecting actions to achieve a specific goal, and descriptive questions regarding the visible properties of objects. We benchmark the performance of existing VideoQA models on two test settings of CRIPP-VQA: in-distribution (ID) and out-of-distribution (OOD), which involve objects with mass, friction coefficient, and initial velocity values that are unseen in the training distribution. In answering questions about the implicit properties (the core focus of this paper) and explicit properties (the focus of prior work) of objects, our experiments uncover a surprisingly substantial performance gap.
