PhysicalAI-Robotics-mindmap-Franka-Mug-in-Drawer

## Dataset Description:

This dataset is a multimodal collection of trajectories generated in Isaac Lab on the ``Mug in Drawer`` task defined in ``mindmap``.

The task was created to evaluate robot manipulation policies on their spatial memory capabilities.
With this (partial) dataset you can generate the full dataset used for [mindmap model](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints/tree/main) training,
run a ``mindmap`` training or evaluate ``mindmap`` open/closed loop.

This dataset is for research and development only.

## Dataset Owner(s):

NVIDIA Corporation

## Dataset Creation Date:

10/15/2025

## License/Terms of Use:

This dataset is governed by the [Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/legalcode)

## Intended Usage:

This dataset is intended for:
Research in robot manipulation policies using imitation learning
Training and evaluating robotic manipulation policies on tasks requiring spatial memory
Training and evaluating the [mindmap model](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints) using the [mindmap codebase](https://github.com/nvidia-isaac/nvblox_mindmap)

## Dataset Characterization:

### Data Collection Method
* Synthetic
* Human teleoperation
* Automatic trajectory generation

15 human teleoperated demonstrations are collected with a SpaceMouse in Isaac Lab.
From these 15 demonstrations, a total of 250 demos are generated automatically using a synthetic motion trajectory generation framework,
[Isaac Lab Mimic](https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html).

## Labeling Method

Not Applicable

## Dataset Format

We provide the Mimic-generated 250 demonstrations in an HDF5 dataset file, and ``mindmap``-formatted datasets for 10 demonstrations converted from the HDF5 file.

Due to storage limitations, we only provide 10 demonstrations in the ``mindmap``-formatted datasets.
If you want to generate the full 250 demos, refer to the ``mindmap`` [data generation docs](https://nvidia-isaac.github.io/nvblox_mindmap/pages/data_generation.html).

The ``mindmap`` dataset consists of tarred demonstration folders. After untarring, the structure is as follows:

```
📂 <DATASET_NAME>/
├── 📂 demo_00000/
│ ├── 00000.<CAMERA_NAME>_depth.png
│ ├── 00000.<CAMERA_NAME>_intrinsics.npy
│ ├── 00000.<CAMERA_NAME>_pose.npy
│ ├── 00000.<CAMERA_NAME>_rgb.png
│ ├── 00000.nvblox_vertex_features.zst
│ ├── 00000.robot_state.npy
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.<CAMERA_NAME>_depth.png
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.robot_state.npy
│ └── demo_successful.npy
├── 📂 demo_00001/
│ └── ...
└── 📂 demo_00009>/
└── dataset_name.hdf5
```

Each demonstration consists of a variable NUM_STEPS with multimodal data:
- RGB-D frame including corresponding camera pose and intrinsics
- Metric-Semantic Reconstruction represented as featurized pointcloud in nvblox_vertex_features.zst
- Robot state including end-effector pose, gripper closedness and head yaw orientation

## Dataset Quantification

Record Count: 250 demonstrations/trajectories

Total Storage: 35 GB

## Reference(s):

- ``mindmap`` paper:
- Remo Steiner, Alexander Millane, David Tingdahl, Clemens Volk, Vikram Ramasamy, Xinjie Yao, Peter Du, Soha Pouya and Shiwei Sheng. "**mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies**". CoRL 2025 Workshop RemembeRL.
[arXiv preprint arXiv:2509.20297 (2025).](https://arxiv.org/abs/2509.20297)
- ``mindmap`` codebase:
- [github.com/nvidia-isaac/nvblox_mindmap](https://github.com/nvidia-isaac/nvblox_mindmap)

## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

数据集描述：
本数据集为多模态轨迹集合，生成于艾萨克实验室（Isaac Lab）中针对mindmap模型定义的「抽屉内杯子任务（Mug in Drawer）」。该任务旨在评估机器人操控策略的空间记忆能力。
借助该（部分）数据集，你可生成用于训练[mindmap模型（mindmap）](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints/tree/main)的完整数据集，开展mindmap模型的训练，或对其开环、闭环性能进行评估。
本数据集仅用于研发用途。

数据集所有者：
英伟达公司（NVIDIA Corporation）

数据集创建日期：
2025年10月15日

使用许可条款：
本数据集受[知识共享署名-非商业性使用4.0国际许可协议（CC BY-NC 4.0）](https://creativecommons.org/licenses/by-nc/4.0/legalcode)约束。

预期用途：
本数据集适用于：
1. 开展基于模仿学习的机器人操控策略研究
2. 在需要空间记忆的任务中训练、评估机器人操控策略
3. 借助[mindmap代码库（mindmap codebase）](https://github.com/nvidia-isaac/nvblox_mindmap)，训练并评估[mindmap模型（mindmap）](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints)

数据集特征描述：
### 数据采集方式
* 合成采集
* 人类遥操作
* 自动轨迹生成
研究人员借助SpaceMouse（SpaceMouse）设备，在艾萨克实验室（Isaac Lab）中采集了15段人类遥操作演示轨迹。基于这15段演示轨迹，研究人员通过合成运动轨迹生成框架艾萨克实验室模拟工具（Isaac Lab Mimic）自动生成了总计250段演示轨迹，相关框架可参阅[https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html](https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html)。

### 标注方式
不适用

### 数据集格式
我们将Mimic生成的250段演示轨迹存储为HDF5数据集文件，并提供了从该HDF5文件转换而来的10段演示轨迹的mindmap格式数据集。
受存储容量限制，本次仅提供10段mindmap格式的演示轨迹数据集。若需生成完整的250段演示轨迹，请参阅mindmap模型的[数据生成文档](https://nvidia-isaac.github.io/nvblox_mindmap/pages/data_generation.html)。
mindmap格式数据集由打包的演示轨迹文件夹组成，解压后的目录结构如下：

📂 <数据集名称>/
├── 📂 demo_00000/
│ ├── 00000.<CAMERA_NAME>_depth.png
│ ├── 00000.<CAMERA_NAME>_intrinsics.npy
│ ├── 00000.<CAMERA_NAME>_pose.npy
│ ├── 00000.<CAMERA_NAME>_rgb.png
│ ├── 00000.nvblox_vertex_features.zst
│ ├── 00000.robot_state.npy
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.<CAMERA_NAME>_depth.png
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.robot_state.npy
│ └── demo_successful.npy
├── 📂 demo_00001/
│ └── ...
└── 📂 demo_00009/
└── dataset_name.hdf5

每段演示轨迹包含可变数量的步数（NUM_STEPS），对应多模态数据如下：
- RGB-D帧，包含对应相机位姿与内参
- 度量语义重建结果，以特征化点云形式存储于nvblox_vertex_features.zst文件中
- 机器人状态数据，包含末端执行器位姿、夹爪开合度与头部偏航角

### 数据集量化统计
记录数量：250段演示轨迹
总存储容量：35 GB

参考资料：
- mindmap模型论文：
Remo Steiner, Alexander Millane, David Tingdahl, Clemens Volk, Vikram Ramasamy, Xinjie Yao, Peter Du, Soha Pouya and Shiwei Sheng. "**mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies**". CoRL 2025 Workshop RemembeRL.
[arXiv预印本arXiv:2509.20297（2025）](https://arxiv.org/abs/2509.20297)
- mindmap模型代码库：
[github.com/nvidia-isaac/nvblox_mindmap](https://github.com/nvidia-isaac/nvblox_mindmap)

伦理考量：
英伟达认为，可信人工智能是一项共同责任，我们已制定相关政策与实践规范，以支持各类人工智能应用的开发。开发者若按照服务条款下载或使用本数据集，应联合内部模型团队，确保该模型符合相关行业与应用场景的要求，并防范未预见的产品误用问题。
请在此处提交安全漏洞报告或英伟达人工智能相关问题反馈：[https://www.nvidia.com/en-us/support/submit-security-vulnerability/](https://www.nvidia.com/en-us/support/submit-security-vulnerability/)