PhysicalAI-Robotics-mindmap-GR1-Drill-in-Box

## Dataset Description:

This dataset is a multimodal collection of trajectories generated in Isaac Lab on the ``Drill in Box`` task defined in ``mindmap``.

The task was created to evaluate robot manipulation policies on their spatial memory capabilities.
With this (partial) dataset you can generate the full dataset used for [mindmap model](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints/tree/main) training,
run a ``mindmap`` training or evaluate ``mindmap`` open/closed loop.

This dataset is for research and development only.

## Dataset Owner(s):

NVIDIA Corporation

## Dataset Creation Date:

10/15/2025

## License/Terms of Use:

This dataset is governed by the [Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/legalcode)

## Intended Usage:

This dataset is intended for:
Research in robot manipulation policies using imitation learning
Training and evaluating robotic manipulation policies on tasks requiring spatial memory
Training and evaluating the [mindmap model](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints) using the [mindmap codebase](https://github.com/nvidia-isaac/nvblox_mindmap)

## Dataset Characterization:

### Data Collection Method
* Synthetic
* Human teleoperation
* Automatic trajectory generation

20 human teleoperated demonstrations are collected through Apple Vision Pro in Isaac Lab.
From these 20 demonstrations, a total of 200 demos are generated automatically using a synthetic motion trajectory generation framework,
[Isaac Lab Mimic](https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html).

## Labeling Method

Not Applicable

## Dataset Format

We provide the Mimic-generated 200 demonstrations in an HDF5 dataset file, and ``mindmap``-formatted datasets for 10 demonstrations converted from the HDF5 file.

Due to storage limitations, we only provide 10 demonstrations in the ``mindmap``-formatted datasets.
If you want to generate the full 200 demos, refer to the ``mindmap`` [data generation docs](https://nvidia-isaac.github.io/nvblox_mindmap/pages/data_generation.html).

The ``mindmap`` dataset consists of tarred demonstration folders. After untarring, the structure is as follows:

```
📂 <DATASET_NAME>/
├── 📂 demo_00000/
│ ├── 00000.<CAMERA_NAME>_depth.png
│ ├── 00000.<CAMERA_NAME>_intrinsics.npy
│ ├── 00000.<CAMERA_NAME>_pose.npy
│ ├── 00000.<CAMERA_NAME>_rgb.png
│ ├── 00000.nvblox_vertex_features.zst
│ ├── 00000.robot_state.npy
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.<CAMERA_NAME>_depth.png
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.robot_state.npy
│ └── demo_successful.npy
├── 📂 demo_00001/
│ └── ...
└── 📂 demo_00009>/
└── dataset_name.hdf5
```

Each demonstration consists of a variable NUM_STEPS with multimodal data:
- RGB-D frame including corresponding camera pose and intrinsics
- Metric-Semantic Reconstruction represented as featurized pointcloud in nvblox_vertex_features.zst
- Robot state including end-effector pose, gripper closedness and head yaw orientation

## Dataset Quantification

Record Count: 200 demonstrations/trajectories

Total Storage: 54 GB

## Reference(s):

- ``mindmap`` paper:
- Remo Steiner, Alexander Millane, David Tingdahl, Clemens Volk, Vikram Ramasamy, Xinjie Yao, Peter Du, Soha Pouya and Shiwei Sheng. "**mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies**". CoRL 2025 Workshop RemembeRL.
[arXiv preprint arXiv:2509.20297 (2025).](https://arxiv.org/abs/2509.20297)
- ``mindmap`` codebase:
- [github.com/nvidia-isaac/nvblox_mindmap](https://github.com/nvidia-isaac/nvblox_mindmap)

## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

## 数据集描述：
本数据集是在Isaac Lab中基于`mindmap`框架定义的「箱内钻孔（Drill in Box）」任务生成的多模态轨迹集合。

该任务旨在评估机器人操控策略的空间记忆能力。借助这份（部分）数据集，你可以生成用于训练[mindmap模型](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints/tree/main)的完整数据集，开展`mindmap`模型训练，或对`mindmap`的开环、闭环性能进行评估。本数据集仅用于研发用途。

## 数据集所有者：
英伟达（NVIDIA）公司

## 数据集创建日期：
2025年10月15日

## 许可与使用条款：
本数据集受[知识共享署名-非商业性使用4.0国际许可协议（CC BY-NC 4.0）](https://creativecommons.org/licenses/by-nc/4.0/legalcode)管辖。

## 适用场景：
本数据集适用于以下场景：
1. 基于模仿学习的机器人操控策略研究
2. 在需要空间记忆的任务上训练与评估机器人操控策略
3. 借助[mindmap代码库](https://github.com/nvidia-isaac/nvblox_mindmap)训练与评估[mindmap模型](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints)

## 数据集特征：
### 数据采集方式
* 合成数据
* 人类遥操作
* 自动轨迹生成

研究人员通过Apple Vision Pro在Isaac Lab中收集了20段人类遥操作演示轨迹。基于这20段演示轨迹，研究人员通过合成运动轨迹生成框架[Isaac Lab Mimic](https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html)自动生成了总计200段演示轨迹。

### 标注方法
不适用

### 数据集格式
我们提供了两种格式的数据：一是由Mimic生成的200段演示轨迹的HDF5数据集文件，二是从该HDF5文件转换而来的、包含10段演示轨迹的`mindmap`格式数据集。受存储容量限制，本次仅提供10段演示轨迹的`mindmap`格式数据集。若需生成完整的200段演示轨迹，请参阅`mindmap`的[数据生成文档](https://nvidia-isaac.github.io/nvblox_mindmap/pages/data_generation.html)。

`mindmap`格式数据集由打包的演示轨迹文件夹组成，解压后的目录结构如下：

📂 <数据集名称>/
├── 📂 demo_00000/
│ ├── 00000.<CAMERA_NAME>_depth.png
│ ├── 00000.<CAMERA_NAME>_intrinsics.npy
│ ├── 00000.<CAMERA_NAME>_pose.npy
│ ├── 00000.<CAMERA_NAME>_rgb.png
│ ├── 00000.nvblox_vertex_features.zst
│ ├── 00000.robot_state.npy
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.<CAMERA_NAME>_depth.png
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.robot_state.npy
│ └── demo_successful.npy
├── 📂 demo_00001/
│ └── ...
└── 📂 demo_00009/
└── dataset_name.hdf5

每段演示轨迹包含可变长度的NUM_STEPS步多模态数据，具体包括：
- 带对应相机位姿与内参的RGB-D帧数据
- 以`nvblox_vertex_features.zst`文件存储的带特征点云形式的度量语义重建结果
- 机器人状态数据，包含末端执行器位姿、夹爪闭合状态与头部偏航角

## 数据集量化指标
记录数量：200段演示轨迹
总存储容量：54 GB

## 参考资料
- `mindmap`相关论文：
- Remo Steiner、Alexander Millane、David Tingdahl、Clemens Volk、Vikram Ramasamy、Xinjie Yao、Peter Du、Soha Pouya及Shiwei Sheng：《**mindmap：面向3D动作策略的深度特征图空间记忆**》，发表于CoRL 2025研讨会RemembeRL。[arXiv预印本arXiv:2509.20297（2025）](https://arxiv.org/abs/2509.20297)
- `mindmap`代码库：[github.com/nvidia-isaac/nvblox_mindmap](https://github.com/nvidia-isaac/nvblox_mindmap)

## 伦理考量
英伟达（NVIDIA）认为，可信人工智能是一项共同责任，我们已制定相关政策与实践规范，以支撑各类人工智能应用的开发。开发者在按照服务条款下载或使用本数据集时，应与内部模型团队协作，确保该模型符合相关行业与应用场景的要求，并防范可能出现的产品误用问题。

请在此处提交安全漏洞或英伟达人工智能相关问题反馈：[https://www.nvidia.com/en-us/support/submit-security-vulnerability/](https://www.nvidia.com/en-us/support/submit-security-vulnerability/)