SPARE3D

包含视图一致性、相机姿态和形状生成三类2D-3D推理任务，难度越来越大。空间推理是人类智能的重要组成部分。我们可以想象 3D 对象的形状并推断它们的空间关系，只需查看它们在 2D 中的三视图线图，具有不同的能力水平。可以训练深度网络来执行空间推理任务吗？我们如何衡量他们的“空间智能”？为了回答这些问题，我们展示了 SPARE3D 数据集。 SPARE3D 基于认知科学和心理测量学，包含视图一致性、相机姿态和形状生成三类 2D-3D 推理任务，难度越来越大。然后，我们设计了一种方法来自动生成大量具有挑战性的问题，并为每个任务提供真实答案。它们用于为使用 ResNet 等最先进的架构训练我们的基线模型提供监督。我们的实验表明，尽管卷积网络在许多视觉学习任务中都取得了超人的表现，但它们在 SPARE3D 任务上的空间推理性能要么低于人类的平均表现，要么甚至接近随机猜测。我们希望 SPARE3D 能够激发空间推理的新问题公式和网络设计，使智能机器人能够通过 2D 传感器在 3D 世界中有效运行。数据集和代码可在 https://ai4ce.github.io/SPARE3D 获得。

This dataset includes three categories of 2D-3D reasoning tasks with increasing difficulty: view consistency, camera pose estimation, and shape generation. Spatial reasoning is a critical component of human intelligence. Humans can imagine the shapes of 3D objects and infer their spatial relationships by merely looking at their 2D line drawings from three orthographic views, with varying levels of proficiency. Can deep networks be trained to perform spatial reasoning tasks? How can we measure their "spatial intelligence"? To answer these questions, we present the SPARE3D dataset. SPARE3D, which is grounded in cognitive science and psychometrics, includes three categories of 2D-3D reasoning tasks with increasing difficulty: view consistency, camera pose estimation, and shape generation. Subsequently, we design a method to automatically generate a large number of challenging questions along with ground-truth answers for each task. These are used to provide supervision for training our baseline models using state-of-the-art architectures such as ResNet. Our experiments demonstrate that although convolutional networks have achieved superhuman performance on many visual learning tasks, their spatial reasoning performance on SPARE3D tasks is either below average human performance or even close to random guessing. We hope that SPARE3D will inspire novel problem formulations and network designs for spatial reasoning, enabling intelligent robots to operate effectively in 3D worlds using 2D sensors. The dataset and code are available at https://ai4ce.github.io/SPARE3D.

搜集汇总

背景与挑战

背景概述

SPARE3D是一个用于评估和提升深度网络在2D-3D空间推理任务中性能的数据集，包含三类难度递增的任务，由纽约大学坦登工程学院于2020年发布。