EmbodiedBench/EB-Man_trajectory_dataset

# EB-Manipulation trajectory dataset We release the trajectory dataset collected from EmbodiedBench using several closed-source and open-source models. We hope this dataset will support the development of more capable embodied agents with improved perception, reasoning, and planning abilities. When using the trajectories, we recommend separating the training and evaluation sets—for example, using the “base” subset for training and other EmbodiedBench subsets for evaluation. ## 📖 Dataset Description (1) eb-man_dataset_multi_step.json. This dataset contains trajectories with multi-step reasoning and planning data (each time the model outputs a plan with multiple actions) for tasks in the EB-Manipulation environment. Each episode provides: - **model_name**: The name of the model used to generate the current trajectory data. - **eval_set**: The evaluation subset the episode belongs to. - **episode_id**: The id of the current episode. - **instruction**: A high-level natural language instruction for the current episode. - **input**: The textual prompt the model takes as input. - **success**: A flag indicating whether the episode was completed successfully (1.0) or not (0.0). - **trajectory** - **visual_description**: Describe the color and shape of each object in the detection box in the numerical order in the image. Then provide the 3D coordinates of the objects chosen from input. - **reasoning_and_reflection**: Reason about the overall plan that needs to be taken on the target objects, and reflect on the previous actions taken if available. - **language_plan**: A list of natural language actions to achieve the user instruction. Each language action is started by the step number and the language action name. - **executable_plan**: A list of discrete actions needed to achieve the user instruction, with each discrete action being a 7-dimensional discrete action. Each action entry contains: - `step_id`: The current step id. - `img_path`: The path to the output image after the current action is executed. - `action`: The 7-dimensional discrete action in the format of a list given by the prompt - `action_success`: 1.0 if that action succeeded, 0.0 if it failed. - `env_feedback`: Environment or simulator feedback, e.g., `"Last action executed successfully."` or error message. - **input_image_path** (`string`): The path to the input image. (2) eb-man_dataset_single_step.json. This dataset is curated from the multi-step version of the trajectory dataset and contains trajectories with single-step reasoning and planning data. --- ## 🔍 Usage You can load the entire dataset in Python as follows: ```python from huggingface_hub import snapshot_download import os import json # 1) Download the entire dataset repo to a local folder local_folder = snapshot_download( repo_id="EmbodiedBench/EB-Man_trajectory_dataset", repo_type="dataset", local_dir="./EB-Man_trajectory_dataset", # or any folder you choose local_dir_use_symlinks=False ) ``` ```python # 2) Load the JSON file with open("./EB-Man_trajectory_dataset/eb-man_dataset_multi_step.json", "r", encoding="utf-8") as f: multi_step_dataset = json.load(f) with open("./EB-Man_trajectory_dataset/eb-man_dataset_single_step.json", "r", encoding="utf-8") as f: single_step_dataset = json.load(f) # Examine the first episode first = multi_step_dataset[0] print("Instruction:", first["instruction"]) print("Number of trajectory steps:", len(first["trajectory"])) ``` You can unzip the images.zip using: ```bash unzip ./EB-Man_trajectory_dataset/images.zip -d ./EB-Man_trajectory_dataset/images ``` ## Citation If you find our dataset helpful for your research, please cite EmbodiedBench: ``` @article{yang2025embodiedbench, title={EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents}, author={Yang, Rui and Chen, Hanyang and Zhang, Junyu and Zhao, Mark and Qian, Cheng and Wang, Kangrui and Wang, Qineng and Koripella, Teja Venkat and Movahedi, Marziyeh and Li, Manling and others}, journal={arXiv preprint arXiv:2502.09560}, year={2025} } ```# EB-Manipulation轨迹数据集 本团队发布了基于EmbodiedBench平台，由多款闭源与开源模型生成的轨迹数据集。我们期望该数据集能够助力具备更优异感知、推理与规划能力的具身智能体（Embodied Agent）的研发工作。在使用该轨迹数据时，建议将训练集与评估集分离——例如，使用“base”子集作为训练集，其余EmbodiedBench子集用于模型评估。 ## 📖 数据集说明 (1) eb-man_dataset_multi_step.json 该数据集包含EB-Manipulation环境下任务的多步推理与规划轨迹数据（即模型每次输出包含多步动作的规划方案）。每个回合包含以下字段： - **model_name**：用于生成当前轨迹数据的模型名称。 - **eval_set**：该回合所属的评估子集。 - **episode_id**：当前回合的唯一标识。 - **instruction**：针对当前回合的高阶自然语言任务指令。 - **input**：模型输入的文本提示词。 - **success**：表征当前回合是否成功完成的标记（1.0代表成功，0.0代表失败）。 - **trajectory** - **visual_description**：按照图像中的数字顺序，描述检测框内每个物体的颜色与外形；随后给出从输入中选定的物体的三维坐标。 - **reasoning_and_reflection**：针对目标物体梳理所需执行的整体规划方案，若存在此前执行过的动作，则对其进行复盘反思。 - **language_plan**：用于完成用户指令的自然语言动作列表。每条自然语言动作均以步骤编号与动作名称作为开头。 - **executable_plan**：用于完成用户指令的离散动作列表，每个离散动作为7维离散动作。每个动作条目包含以下内容： - `step_id`：当前步骤的编号。 - `img_path`：当前动作执行完成后输出图像的存储路径。 - `action`：以提示词规定的列表格式呈现的7维离散动作。 - `action_success`：表征该动作是否执行成功的标记（1.0为成功，0.0为失败）。 - `env_feedback`：环境或模拟器返回的反馈信息，例如`"Last action executed successfully."`或错误提示。 - **input_image_path**（`string`类型）：输入图像的存储路径。 (2) eb-man_dataset_single_step.json 该数据集从多步轨迹数据集中精简得到，仅包含单步推理与规划的轨迹数据。 --- ## 🔍 使用方法 您可通过如下Python代码加载完整数据集： python from huggingface_hub import snapshot_download import os import json # 1) Download the entire dataset repo to a local folder local_folder = snapshot_download( repo_id="EmbodiedBench/EB-Man_trajectory_dataset", repo_type="dataset", local_dir="./EB-Man_trajectory_dataset", # or any folder you choose local_dir_use_symlinks=False ) python # 2) Load the JSON file with open("./EB-Man_trajectory_dataset/eb-man_dataset_multi_step.json", "r", encoding="utf-8") as f: multi_step_dataset = json.load(f) with open("./EB-Man_trajectory_dataset/eb-man_dataset_single_step.json", "r", encoding="utf-8") as f: single_step_dataset = json.load(f) # Examine the first episode first = multi_step_dataset[0] print("Instruction:", first["instruction"]) print("Number of trajectory steps:", len(first["trajectory"])) 您可通过如下命令解压images.zip压缩包： bash unzip ./EB-Man_trajectory_dataset/images.zip -d ./EB-Man_trajectory_dataset/images ## 引用说明 若本数据集对您的研究工作有所助益，请引用EmbodiedBench相关论文： @article{yang2025embodiedbench, title={EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents}, author={Yang, Rui and Chen, Hanyang and Zhang, Junyu and Zhao, Mark and Qian, Cheng and Wang, Kangrui and Wang, Qineng and Koripella, Teja Venkat and Movahedi, Marziyeh and Li, Manling and others}, journal={arXiv preprint arXiv:2502.09560}, year={2025} }