J-CRe3

J-CRe3是一个专为真实世界参考解析设计的日语对话数据集，由京都大学和RIKEN合作创建。该数据集包含93个视频和对话音频，记录了家庭环境中主人和助理机器人之间的真实对话，涉及大量物体操作任务。数据集通过众包方式收集对话场景，并由演员模拟对话录制。每个对话都与视频帧中的物体边界框进行了跨模态标注，包括直接和间接参考关系。J-CRe3旨在解决机器人与人类在真实环境中交互时的理解问题，特别是在处理零参考（即省略的指代）时。J-CRe3 is a Japanese dialogue dataset dedicated to real-world reference resolution, co-created by Kyoto University and RIKEN. The dataset comprises 93 sets of video and paired dialogue audio recordings, capturing authentic dialogues between a human homeowner and an assistive robot in a home environment, involving a substantial number of object manipulation tasks. The dialogue scenarios were collected via crowdsourcing, and the recordings were produced by actors simulating the interactive sessions. Each dialogue is cross-modally annotated with object bounding boxes in the corresponding video frames, covering both direct and indirect reference relations. J-CRe3 aims to address the understanding challenges in human-robot interaction within real-world environments, particularly when handling zero references, i.e., omitted referential expressions.

搜集汇总

背景与挑战

背景概述

J-CRe3是一个用于真实世界参考解析的日语对话数据集，包含93个场景对话、2,131个话语和超过11,000秒的视频数据。数据集提供多模态标注，如边界框、文本引用和文本到对象引用，支持自然语言处理和计算机视觉任务，并遵循CC BY-SA 4.0许可。