CAESAR: An Embodied Simulator for Generating Multimodal Referring Expression Datasets

Humans naturally use verbal utterances and nonverbal gestures to refer to various objects (known as referring expressions) in different interactional scenarios. As collecting real human interaction datasets are costly and laborious, synthetic datasets are often used to train models to unambiguously detect relationships among objects. However, existing synthetic data generation tools that provide referring expressions generally neglect nonverbal gestures. Additionally, while a few small-scale datasets contain multimodal cues (verbal and nonverbal), these datasets only capture the nonverbal gestures from an exo-centric perspective (observer). As models can use complementary information from multimodal cues to recognize referring expressions, generating multimodal data from multiple views can help to develop robust models. To address these critical issues, in this paper, we present a novel embodied simulator, CAESAR, to generate multimodal referring expressions containing both verbal utterances and nonverbal cues captured from multiple views. Using our simulator, we have generated two large-scale embodied referring expression datasets, which we will release publicly. We have conducted experimental analyses on embodied spatial relation grounding using various state-of-the-art baseline models. Our experimental results suggest that visual perspective affects the models' performance; and that nonverbal cues improve spatial relation grounding accuracy. Finally, we will release the simulator publicly to allow researchers to generate new embodied interaction datasets.

人类在各类交互场景中，通常会借助语言话语与非语言手势来指代各类对象，此类指代方式被称为指代表达式（referring expressions）。采集真实人类交互数据集不仅成本高昂，且耗时费力，研究人员常采用合成数据集训练模型，以精准识别对象间的关联关系。然而，现有可生成指代表达式的合成数据生成工具，普遍忽略了非语言手势这一关键模态。此外，尽管少数小型数据集涵盖了语言与非语言两类多模态线索，但这类数据集仅从外中心视角（即观察者视角）采集非语言手势数据。由于模型可借助多模态线索的互补信息识别指代表达式，从多视角生成多模态数据有助于构建鲁棒性更强的模型。为解决上述核心问题，本文提出一款全新的具身模拟器（embodied simulator）CAESAR，可生成包含语言话语与多视角采集的非语言线索的多模态指代表达式。依托该模拟器，我们构建了两套大规模具身指代表达式数据集，并将对外公开发布。我们采用多款当前最先进的基线模型，针对具身空间关系锚定任务开展了实验分析。实验结果表明，视觉视角会对模型性能产生显著影响，而非语言线索则可有效提升空间关系锚定的准确率。最后，我们将对外公开发布该模拟器，以供研究人员生成全新的具身交互数据集。

搜集汇总

背景与挑战

背景概述

CAESAR是一个用于生成多模态指代表达数据集的具身模拟器，它结合了言语表达和非语言手势，并从多个视角捕捉数据，以弥补现有合成数据工具在非语言手势和多视角数据方面的不足。该模拟器已生成两个大规模公开数据集，实验表明非语言线索能提高空间关系理解的准确性，且工具将公开供研究者生成新的交互数据集。