TIDEE

我们介绍了TIDEE，这是一种具体化的代理，可以根据所学到的常识性对象放置和房间布置先验来整理无序的场景。TIDEE探索家庭环境，检测不在其自然位置的对象，为其推断出合理的对象上下文，在当前场景中定位此类上下文，并重新定位对象。常识先验被编码为三个模块: i) 视觉语义检测器，用于检测不适当的对象; ii) 对象和空间关系的关联神经图存储器，该存储器提出了用于对象重新定位的合理语义容器和表面，iii) 视觉搜索网络，可指导代理的探索，以有效地定位当前场景中的感兴趣的容器以重新定位对象。我们测试TIDEE在AI2THOR模拟环境中整理混乱的场景。
数据集包含8000训练场景、200验证场景和100测试场景，其中每个场景中的五个对象被移出原位。

We introduce TIDEE, an embodied agent that tidies unordered scenes based on learned prior knowledge of common-sense object placement and room layout. TIDEE explores household environments, detects objects that are not in their natural positions, infers plausible object contexts for them, locates such contexts in the current scene, and relocates the objects. The common-sense priors are encoded into three modules: i) a visual semantic detector for identifying misplaced objects; ii) an associative neural graph memory for modeling object and spatial relationships, which proposes plausible semantic containers and surfaces for object relocation; iii) a visual search network that guides the agent’s exploration to efficiently locate the target containers in the current scene for object relocation. We evaluate TIDEE on tidying cluttered scenes in the AI2THOR simulated environment. The dataset consists of 8000 training scenes, 200 validation scenes, and 100 test scenes, where five objects in each scene are moved out of their original positions.

搜集汇总

背景与挑战

背景概述

TIDEE是一个基于常识先验的具身智能代理数据集，包含8300个模拟家庭场景(含训练/验证/测试集)，每个场景有5个位置错乱的对象，用于研究智能代理整理杂乱环境的能力。该系统通过视觉检测、神经图记忆和视觉搜索三大模块实现对象重新定位。