EmbodiedBench/EB-Habitat_trajectory_dataset

# EB-Habitat trajectory dataset We release the trajectory dataset collected from EmbodiedBench using several closed-source and open-source models. We hope this dataset will support the development of more capable embodied agents with improved perception, reasoning, and planning abilities. When using the trajectories, we recommend separating the training and evaluation sets—for example, using the “base” subset for training and other EmbodiedBench subsets for evaluation. ## 📖 Dataset Description (1) eb-habitat_dataset_multi_step.json. This dataset contains trajectories with multi-step reasoning and planning data (each time the model outputs a plan with multiple actions) for tasks in the EB-Habitat environment. Each episode provides: - **model_name**: The name of the model used to generate the current trajectory data. - **eval_set**: The evaluation subset the episode belongs to. - **episode_id**: The id of the current episode. - **instruction**: A high-level natural language instruction for the current episode. - **input**: The textual prompt the model takes as input. - **success**: A flag indicating whether the episode was completed successfully (1.0) or not (0.0). - **trajectory** - **visual_description**: Description of current state from the visual image. - **reasoning_and_reflection**: summarize the history of interactions and any available environmental feedback. Additionally, provide reasoning as to why the last action or plan failed and did not finish the task. - **language_plan**: The list of actions to achieve the user instruction. Each action is started by the step number and the action name. - **executable_plan**: A list of actions needed to achieve the user instruction, with each action having an action ID and a name. Each action entry contains: - `step_id`: The current step id. - `img_path`: The path to the output image after the current action is executed. - `action` (`[id, name]`): A two-element list where the first element is the action ID and the second is the action name. - `action_success`: 1.0 if that action succeeded, 0.0 if it failed. - `env_feedback`: Environment or simulator feedback, e.g., `"Last action executed successfully."` or error message. - **input_image_path** (`string`): The path to the input image. (2) eb-habitat_dataset_single_step.json. This dataset is curated from the multi-step version of the trajectory dataset and contains trajectories with single-step reasoning and planning data. --- ## 🔍 Usage You can load the entire dataset in Python as follows: ```python from huggingface_hub import snapshot_download import os import json # 1) Download the entire dataset repo to a local folder local_folder = snapshot_download( repo_id="EmbodiedBench/EB-Habitat_trajectory_dataset", repo_type="dataset", local_dir="./EB-Habitat_trajectory_dataset", # or any folder you choose local_dir_use_symlinks=False ) ``` ```python # 2) Load the JSON file with open("./EB-Habitat_trajectory_dataset/eb-habitat_dataset_multi_step.json", "r", encoding="utf-8") as f: multi_step_dataset = json.load(f) with open("./EB-Habitat_trajectory_dataset/eb-habitat_dataset_single_step.json", "r", encoding="utf-8") as f: single_step_dataset = json.load(f) # Examine the first episode first = multi_step_dataset[0] print("Instruction:", first["instruction"]) print("Number of trajectory steps:", len(first["trajectory"])) ``` You can unzip the images.zip using: ```bash unzip ./EB-Habitat_trajectory_dataset/images.zip -d ./EB-Habitat_trajectory_dataset/images ``` ## Citation If you find our dataset helpful for your research, please cite EmbodiedBench: ``` @article{yang2025embodiedbench, title={EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents}, author={Yang, Rui and Chen, Hanyang and Zhang, Junyu and Zhao, Mark and Qian, Cheng and Wang, Kangrui and Wang, Qineng and Koripella, Teja Venkat and Movahedi, Marziyeh and Li, Manling and others}, journal={arXiv preprint arXiv:2502.09560}, year={2025} } ```# EB-Habitat轨迹数据集 我们发布了基于EmbodiedBench采集的轨迹数据集，该数据集由多款闭源与开源模型生成。我们期望本数据集能够助力研发具备更优异感知、推理与规划能力的具身智能体（AI Agent）。在使用该轨迹数据集时，建议拆分训练集与评估集——例如，以"base"子集作为训练集，其余EmbodiedBench子集作为评估集。 ## 📖 数据集说明 (1) eb-habitat_dataset_multi_step.json 本数据集包含EB-Habitat环境下各类任务的轨迹数据，涵盖多步推理与规划内容（即模型每次输出包含多步动作的规划方案）。每个任务回合（episode）包含以下字段： - **model_name**：用于生成当前轨迹数据的模型名称。 - **eval_set**：当前任务回合所属的评估子集。 - **episode_id**：当前任务回合的唯一标识。 - **instruction**：当前任务回合的高层级自然语言指令。 - **input**：模型接收的文本提示输入。 - **success**：任务完成状态标识，成功为1.0，失败为0.0。 - **trajectory** 轨迹详情 - **visual_description**：基于当前视觉图像生成的场景状态描述。 - **reasoning_and_reflection**：对交互历史与可用环境反馈的总结，同时需推理说明上一步动作或规划失败且未完成任务的原因。 - **language_plan**：达成用户指令所需的动作列表，每个动作以步骤编号与动作名称作为开头。 - **executable_plan**：达成用户指令所需的可执行动作列表，每个动作包含动作ID与名称。每个动作条目包含： - `step_id`：当前步骤编号。 - `img_path`：当前动作执行后输出图像的存储路径。 - `action` (`[id, name]`)：二元列表，第一个元素为动作ID，第二个元素为动作名称。 - `action_success`：动作执行成功则为1.0，失败则为0.0。 - `env_feedback`：环境或模拟器返回的反馈，例如"Last action executed successfully."或错误信息。 - **input_image_path** (`string`)：输入图像的存储路径。 (2) eb-habitat_dataset_single_step.json 本数据集由多步版本轨迹数据集精简而来，仅包含单步推理与规划数据的轨迹内容。 --- ## 🔍 使用方法 你可通过如下Python代码加载完整数据集： python from huggingface_hub import snapshot_download import os import json # 1) 将完整数据集仓库下载至本地文件夹 local_folder = snapshot_download( repo_id="EmbodiedBench/EB-Habitat_trajectory_dataset", repo_type="dataset", local_dir="./EB-Habitat_trajectory_dataset", # 或自定义任意文件夹路径 local_dir_use_symlinks=False ) python # 2) 加载JSON格式数据集文件 with open("./EB-Habitat_trajectory_dataset/eb-habitat_dataset_multi_step.json", "r", encoding="utf-8") as f: multi_step_dataset = json.load(f) with open("./EB-Habitat_trajectory_dataset/eb-habitat_dataset_single_step.json", "r", encoding="utf-8") as f: single_step_dataset = json.load(f) # 查看首个任务回合 first = multi_step_dataset[0] print("指令内容:", first["instruction"]) print("轨迹步骤总数:", len(first["trajectory"])) 你可通过以下命令解压images.zip压缩包： bash unzip ./EB-Habitat_trajectory_dataset/images.zip -d ./EB-Habitat_trajectory_dataset/images ## 📝 引用说明 若本数据集对你的研究有所帮助，请引用EmbodiedBench相关论文： @article{yang2025embodiedbench, title={EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents}, author={Yang, Rui and Chen, Hanyang and Zhang, Junyu and Zhao, Mark and Qian, Cheng and Wang, Kangrui and Wang, Qineng and Koripella, Teja Venkat and Movahedi, Marziyeh and Li, Manling and others}, journal={arXiv preprint arXiv:2502.09560}, year={2025} }