PhysicalAI-GR00T-Tuned-Tasks

## Dataset Description:

This dataset is multimodal collections of trajectories generated in Isaac Lab. It supports humanoid (GR1) tabletop manipulation tasks for industrial settings. Each dataset entry provides the full context (state, vision, language, action) needed to train and evaluate generalist robot policies for tasks like pouring nuts or sorting pipes by color.

| Dataset Name | # Trajectories |
|---------------------------|----------------|
| Exhaust-Pipe-Sorting-task | 1000 |
| Nut-Pouring-task | 1000 |

This dataset is ideal for behavior cloning, policy learning, and generalist robotic manipulation research. It has been for post training GR00T N1 model.

This dataset is ready for commercial use.

## Dataset Owner
NVIDIA Corporation

## Dataset Creation Date:
05/09/2025

## License/Terms of Use:
This dataset is governed by the Creative Commons Attribution 4.0 International License (CC-BY-4.0).

## Intended Usage:
This dataset is intended for:

- Training robot manipulation policies using behavior cloning.
- Research in generalist robotics and task-conditioned agents.
- Sim-to-real / Sim-to-Sim transfer studies.

## Dataset Characterization:
### Data Collection Method

- Automated
- Automatic/Sensors
- Synthetic

5 human teleoperated demonstrations are collected through Apple Vision Pro in Isaac Lab. All 1,000 demos are generated automatically using a synthetic motion trajectory generation framework, Mimicgen [1]. Each demo is generated at 20 Hz.

### Labeling Method

Not Applicable

## Dataset Format:
We provide the Mimic-generated 1000 demonstrations in HDF5 dataset files, and GR00T-Lerobot formatted datasets converted from HDF5 files.
Each demo in GR00T-Lerobot datasets consists of a time-indexed sequence of the following modalities:

### Actions
- action (FP64): joint desired positions for all body joints (26 DoF)

### Observations
- observation.state (FP64): joint positions for all body joints (26 DoF)

### Task-specific
- timestamp (FP64): simulation time in seconds of each recorded data entry.
- annotation.human.action.task_description (INT64): index referring to the language instruction recorded in the metadata
- annotation.human.action.valid (INT64): index indicating validity of annotaion recorded in the metadata
- episode_index (INT64): index indicating the order of each demo
- task_index (INT64): index used in multi-task data loader. Not applicable to Gr00t-N1 post training, always set to 0.

### Videos
- 256 x 256 RGB videos in mp4 format from first-person-view camera

In additional, a set of metadata describing the followings is provided,
- `episodes.jsonl` contains a list of all the episodes in the entire dataset. Each episode contains a list of tasks and the length of the episode.
- `tasks.jsonl` contains a list of all the tasks in the entire dataset.
- `modality.json` contains the modality configuration.
- `info.json` contains the dataset information.

## Dataset Quantification:

### Record Count

#### Exhaust Pipe Sorting Task
- Number of demonstrations/trajectories: 1000
- Number of RGB videos: 1000

#### Nut Pouring Task
- Number of demonstrations/trajectories: 1000
- Number of RGB videos: 1000

### Total Storage

26.5 GB

## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

## Reference(s):
[1] @inproceedings{mandlekar2023mimicgen,
title={MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations},
author={Mandlekar, Ajay and Nasiriany, Soroush and Wen, Bowen and Akinola, Iretiayo and Narang, Yashraj and Fan, Linxi and Zhu, Yuke and Fox, Dieter},
booktitle={7th Annual Conference on Robot Learning},
year={2023}
}

## 数据集描述

本数据集为艾萨克实验室（Isaac Lab）中生成的多模态轨迹集合，支持适用于工业场景的类人型（GR1）桌面操作任务。每条数据集条目均包含训练与评估通用型机器人策略所需的完整上下文信息（包括状态、视觉、语言与动作），可用于坚果倾倒、按颜色分拣排气管等任务。

| 数据集名称 | 轨迹数量 |
|---------------------------|--------|
| 排气管分拣任务（Exhaust-Pipe-Sorting-task） | 1000 |
| 坚果倾倒任务（Nut-Pouring-task） | 1000 |

本数据集适用于行为克隆、策略学习以及通用机器人操作研究，已用于GR00T N1模型的后续训练。本数据集可用于商业场景。

## 数据集所有者
英伟达公司（NVIDIA Corporation）

## 数据集创建日期
2025年5月9日

## 使用许可协议
本数据集受知识共享署名4.0国际许可协议（CC-BY-4.0）管辖。

## 预期用途
本数据集适用于以下场景：
- 使用行为克隆训练机器人操作策略
- 通用机器人学与任务条件智能体研究
- 仿真到真实（Sim-to-real）/仿真到仿真（Sim-to-Sim）迁移研究

## 数据集特征
### 数据采集方式
- 自动化采集
- 自动化/传感器采集
- 合成数据

通过苹果Vision Pro（Apple Vision Pro）在艾萨克实验室（Isaac Lab）中采集了5条人工遥操作演示轨迹；剩余全部1000条演示轨迹均通过合成运动轨迹生成框架Mimicgen[1]自动生成，每条轨迹的采样频率为20Hz。

### 标注方式
不适用

## 数据集格式
我们提供了Mimicgen生成的1000条演示轨迹的HDF5格式数据集文件，以及从HDF5文件转换而来的GR00T-Lerobot格式数据集。
GR00T-Lerobot格式数据集中的每条演示轨迹均由按时间索引的以下多模态数据组成：

### 动作数据
- "action"（FP64）：所有身体关节的期望关节位置（共26个自由度）

### 观测数据
- "observation.state"（FP64）：所有身体关节的实际关节位置（共26个自由度）

### 任务特定信息
- "timestamp"（FP64）：每条记录数据的仿真时间，单位为秒
- "annotation.human.action.task_description"（INT64）：指向元数据中记录的语言指令的索引
- "annotation.human.action.valid"（INT64）：指向元数据中记录的标注有效性的索引
- "episode_index"（INT64）：指示每条演示轨迹顺序的索引
- "task_index"（INT64）：多任务数据加载器中使用的索引。本数据集用于GR00T-N1后续训练时不适用，固定设为0。

### 视频数据
- 来自第一视角摄像头的256×256分辨率mp4格式RGB视频

此外还提供了描述以下内容的元数据文件：
- `episodes.jsonl`：包含整个数据集中所有演示轨迹的列表，每条轨迹包含对应任务列表与轨迹长度
- `tasks.jsonl`：包含整个数据集中所有任务的列表
- `modality.json`：包含数据集的模态配置信息
- `info.json`：包含数据集的相关信息

## 数据集量化统计
### 记录统计
#### 排气管分拣任务
- 演示轨迹/轨迹数量：1000
- RGB视频数量：1000

#### 坚果倾倒任务
- 演示轨迹/轨迹数量：1000
- RGB视频数量：1000

### 总存储容量
26.5 GB

## 伦理考量
英伟达（NVIDIA）认为，可信人工智能是一项共同责任，我们已制定相关政策与实践规范，以支持各类人工智能应用的开发。开发者在按照本服务条款下载或使用本数据集时，应与其内部模型团队协作，确保本模型符合相关行业与应用场景的要求，并应对可能出现的产品误用问题。

请通过[此链接](https://www.nvidia.com/en-us/support/submit-security-vulnerability/)报告安全漏洞或英伟达人工智能相关问题。

## 参考文献
[1] @inproceedings{mandlekar2023mimicgen,
title={MimicGen: 基于人工演示的可扩展机器人学习数据生成系统},
author={Mandlekar, Ajay and Nasiriany, Soroush and Wen, Bowen and Akinola, Iretiayo and Narang, Yashraj and Fan, Linxi and Zhu, Yuke and Fox, Dieter},
booktitle={第7届机器人学习年度会议},
year={2023}
}