thomas-yanxin/robospatial-home-mirror

--- license: apache-2.0 dataset_info: features: - name: category dtype: string - name: question dtype: string - name: answer dtype: string - name: img dtype: image - name: depth_image dtype: image - name: mask dtype: image splits: - name: context num_bytes: 38727163.0 num_examples: 122 - name: compatibility num_bytes: 32578930.0 num_examples: 105 - name: configuration num_bytes: 37179831.0 num_examples: 123 download_size: 34740838 dataset_size: 108485924.0 configs: - config_name: default data_files: - split: context path: data/context-* - split: compatibility path: data/compatibility-* - split: configuration path: data/configuration-* task_categories: - question-answering - visual-question-answering language: - en pretty_name: robospatial-home size_categories: - n<1K --- # RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics [**🌐 Homepage**](https://chanh.ee/RoboSpatial/) | [**📖 arXiv**](https://arxiv.org/abs/2411.16537) | [**🛠️ Data Gen**](https://github.com/NVlabs/RoboSpatial) | [**🧪 Eval Code**](https://github.com/chanhee-luke/RoboSpatial-Eval) ## 🔥 **Core spatial understanding benchmark used by [Qwen3-VL](https://qwen.ai/blog?id=99f0335c4ad9ff6153e517418d48535ab6d8afef&from=research.latest-advancements-list) and [Gemini Robotics](https://storage.googleapis.com/deepmind-media/gemini-robotics/Gemini-Robotics-1-5-Tech-Report.pdf)!** ## ⚠️ Important Note (08/14/25) **Annotation Correction:** In the context category, the microwave question was corrected from “below” to “above” to fix an annotation error. ## Dataset Description We introduce RoboSpatial-Home: a new spatial reasoning benchmark designed to evaluate vision-language models (VLMs) in real-world indoor environments for robotics. It consists of 350 spatial reasoning questions paired with crowd-sourced RGBD images captured using a handheld iPhone camera equipped with a depth sensor. Each image is annotated with three types of spatial relationship questions—spatial configuration, spatial context, and spatial compatibility—providing a comprehensive evaluation of spatial understanding in robotic applications. ## Dataset Structure RoboSpatial-Home consists of QA annotations paired with RGB and depth images. The dataset is organized as follows: - `category`: The spatial reasoning category for the entry (configuration, context, or compatibility). - `question`: The spatial reasoning question. - `answer`: The human annotated answer. - `img`: The RGB image from iPhone 13 Pro Max. - `depth_image`: The corresponding depth image from iPhone 13 Pro Max. - `mask`: (If available) The corresponding segmentation mask for spatial context questions. ## QA types: - Spatial Configuration: Determines the relative positioning of objects (e.g., "*Is the mug to the left of the laptop?*"). - Spatial Context: Identifies vacant areas in relation to a reference object (e.g., "*Identify empty space to the left of the bowl.*"). - Spatial Compatibility: Assesses whether an object can fit within a specified area (e.g., "*Can the chair be placed in front of the desk?*"). ## Load Dataset You can load the RoboSpatial-Home dataset in two ways: 1. Using the Hugging Face `datasets` library ```python import datasets dataset_name = 'chanhee-luke/RoboSpatial-Home' data = load_dataset(dataset_name, CATEGORY) ``` where `CATEGORY` is one of the spatial reasoning categories: `configuration`, `context`, `compatibility`. If not specified, the entire dataset will be loaded. 2. Downloading locally with the script If you prefer to work with local files, the RoboSpatial-Eval repo provides a [script](https://github.com/chanhee-luke/RoboSpatial-Eval/blob/master/download_benchmark.py): ``` python download_benchmark.py [OUTPUT_FOLDER_PATH] ``` This downloads the dataset locally for debugging or for setups that don’t use the Hugging Face `datasets` library. If no output path is provided, the dataset will be saved to `./RoboSpatial-Home` by default. ## Dataset Creation The data for RoboSpatial-Home was manually collected and annotated by graduate-level students in computer science. ## Disclaimers ⚠️ Disclaimer: The images in this dataset were collected from real homes of real individuals. When using or distributing this dataset, ensure that privacy and ethical considerations are upheld. Redistribution of images should be done with caution to respect the privacy of the original contributors. ## Contact - Luke Song: [e-mail] ## Citation **BibTex:** ```bibtex @inproceedings{song2025robospatial, author = {Song, Chan Hee and Blukis, Valts and Tremblay, Jonathan and Tyree, Stephen and Su, Yu and Birchfield, Stan}, title = {{RoboSpatial}: Teaching Spatial Understanding to {2D} and {3D} Vision-Language Models for Robotics}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2025}, note = {Oral Presentation}, } ```许可证：Apache-2.0 数据集信息： 特征字段： - 字段名：category，数据类型：字符串 - 字段名：question，数据类型：字符串 - 字段名：answer，数据类型：字符串 - 字段名：img，数据类型：图像 - 字段名：depth_image，数据类型：深度图像 - 字段名：mask，数据类型：掩码图像 数据拆分： - 拆分名称：context，字节数：38727163.0，样本数：122 - 拆分名称：compatibility，字节数：32578930.0，样本数：105 - 拆分名称：configuration，字节数：37179831.0，样本数：123 下载大小：34740838 数据集总大小：108485924.0 配置项： - 配置名称：默认 数据文件： - 拆分：context，路径：data/context-* - 拆分：compatibility，路径：data/compatibility-* - 拆分：configuration，路径：data/configuration-* 任务类别： - 问答 - 视觉问答 语言： - 英语 展示名：robospatial-home 样本规模分类： - 样本数少于1000 --- # RoboSpatial：面向机器人学的2D与3D视觉语言模型空间理解教学数据集 [**🌐 项目主页**](https://chanh.ee/RoboSpatial/) | [**📖 arXiv论文**](https://arxiv.org/abs/2411.16537) | [**🛠️ 数据生成代码**](https://github.com/NVlabs/RoboSpatial) | [**🧪 评估代码**](https://github.com/chanhee-luke/RoboSpatial-Eval) ## 🔥 **被[Qwen3-VL](https://qwen.ai/blog?id=99f0335c4ad9ff6153e517418d48535ab6d8afef&from=research.latest-advancements-list)与[Gemini Robotics](https://storage.googleapis.com/deepmind-media/gemini-robotics/Gemini-Robotics-1-5-Tech-Report.pdf)采用的核心空间理解基准数据集！** ## ⚠️ 重要说明（2025年8月14日） **标注修正：在context类别中，微波炉相关问题的描述已从“下方”修正为“上方”，以修复标注错误。** ## 数据集简介 我们提出RoboSpatial-Home：一款全新的空间推理基准数据集，用于评估面向真实室内机器人场景的视觉语言模型（Vision-Language Model, VLM）。该数据集包含350条空间推理问题，搭配由搭载深度传感器的手持iPhone相机采集的众包RGBD图像。每张图像均标注了三类空间关系问题：空间配置、空间上下文与空间兼容性，可全面评估机器人应用中的空间理解能力。 ## 数据集结构 RoboSpatial-Home由问答标注与RGB、深度图像组成，数据集结构如下： - `category`：样本所属的空间推理类别（配置、上下文或兼容性） - `question`：空间推理问题文本 - `answer`：人工标注的标准答案 - `img`：iPhone 13 Pro Max拍摄的RGB图像 - `depth_image`：对应iPhone 13 Pro Max拍摄的深度图像 - `mask`：（若存在）空间上下文问题对应的分割掩码 ## 问答类型： - 空间配置：判断物体间的相对位置关系（例如：“杯子是否在笔记本电脑左侧？”）。 - 空间上下文：识别参考物体周边的空余区域（例如：“识别碗左侧的空区域。”）。 - 空间兼容性：评估某一物体是否可放置于指定区域内（例如：“椅子能否放置在书桌前方？”）。 ## 数据集加载 可通过两种方式加载RoboSpatial-Home数据集： 1. 使用Hugging Face `datasets`库 python import datasets dataset_name = 'chanhee-luke/RoboSpatial-Home' data = load_dataset(dataset_name, CATEGORY) 其中`CATEGORY`为以下三类空间推理类别之一：`configuration`、`context`、`compatibility`。若未指定类别，则加载完整数据集。 2. 使用脚本本地下载 若偏好使用本地文件，RoboSpatial-Eval仓库提供了[下载脚本](https://github.com/chanhee-luke/RoboSpatial-Eval/blob/master/download_benchmark.py)： python download_benchmark.py [OUTPUT_FOLDER_PATH] 该脚本可将数据集下载至本地，用于调试或不使用Hugging Face `datasets`库的场景。若未指定输出路径，数据集将默认保存至`./RoboSpatial-Home`。 ## 数据集构建 RoboSpatial-Home的数据由计算机科学方向的研究生手动采集并标注。 ## 免责声明 ⚠️ 免责声明：本数据集的图像采集自真实个体的住宅。在使用或分发该数据集时，请务必遵守隐私与伦理规范。分发图像时需谨慎处理，以尊重原始贡献者的隐私。 ## 联系方式 - Luke Song: [e-mail] ## 引用格式 **BibTex:** bibtex @inproceedings{song2025robospatial, author = {Song, Chan Hee and Blukis, Valts and Tremblay, Jonathan and Tyree, Stephen and Su, Yu and Birchfield, Stan}, title = {{RoboSpatial}: Teaching Spatial Understanding to {2D} and {3D} Vision-Language Models for Robotics}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2025}, note = {Oral Presentation}, }