PhysicalAI-Robotics-mindmap-Franka-Cube-Stacking

## Dataset Description:

This dataset is a multimodal collection of trajectories generated in Isaac Lab on the ``Cube Stacking`` task defined in ``mindmap``.

The task was created to evaluate robot manipulation policies on their spatial memory capabilities.
With this (partial) dataset you can generate the full dataset used for [mindmap model](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints/tree/main) training,
run a ``mindmap`` training or evaluate ``mindmap`` open/closed loop.

This dataset is for research and development only.

## Dataset Owner(s):

NVIDIA Corporation

## Dataset Creation Date:

10/15/2025

## License/Terms of Use:

This dataset is governed by the [Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/legalcode)

## Intended Usage:

This dataset is intended for:
Research in robot manipulation policies using imitation learning
Training and evaluating robotic manipulation policies on tasks requiring spatial memory
Training and evaluating the [mindmap model](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints) using the [mindmap codebase](https://github.com/nvidia-isaac/nvblox_mindmap)

## Dataset Characterization:

### Data Collection Method
* Synthetic
* Human teleoperation
* Automatic trajectory generation

10 human teleoperated demonstrations are collected with a SpaceMouse in Isaac Lab.
From these 10 demonstrations, a total of 1000 demos are generated automatically using a synthetic motion trajectory generation framework,
[Isaac Lab Mimic](https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html).

## Labeling Method

Not Applicable

## Dataset Format

We provide the Mimic-generated 1000 demonstrations in an HDF5 dataset file, and ``mindmap``-formatted datasets for 10 demonstrations converted from the HDF5 file.

Due to storage limitations, we only provide 10 demonstrations in the ``mindmap``-formatted datasets.
If you want to generate the full 1000 demos, refer to the ``mindmap`` [data generation docs](https://nvidia-isaac.github.io/nvblox_mindmap/pages/data_generation.html).

The ``mindmap`` dataset consists of tarred demonstration folders. After untarring, the structure is as follows:

```
📂 <DATASET_NAME>/
├── 📂 demo_00000/
│ ├── 00000.<CAMERA_NAME>_depth.png
│ ├── 00000.<CAMERA_NAME>_intrinsics.npy
│ ├── 00000.<CAMERA_NAME>_pose.npy
│ ├── 00000.<CAMERA_NAME>_rgb.png
│ ├── 00000.nvblox_vertex_features.zst
│ ├── 00000.robot_state.npy
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.<CAMERA_NAME>_depth.png
│ ├── ...
│ ├── <NUM_STEPS_IN_DEMO>.robot_state.npy
│ └── demo_successful.npy
├── 📂 demo_00001/
│ └── ...
└── 📂 demo_00009>/
└── dataset_name.hdf5
```

Each demonstration consists of a variable NUM_STEPS with multimodal data:
- RGB-D frame including corresponding camera pose and intrinsics
- Metric-Semantic Reconstruction represented as featurized pointcloud in nvblox_vertex_features.zst
- Robot state including end-effector pose, gripper closedness and head yaw orientation

## Dataset Quantification

Record Count: 1000 demonstrations/trajectories

Total Storage: 6 GB

## Reference(s):

- ``mindmap`` paper:
- Remo Steiner, Alexander Millane, David Tingdahl, Clemens Volk, Vikram Ramasamy, Xinjie Yao, Peter Du, Soha Pouya and Shiwei Sheng. "**mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies**". CoRL 2025 Workshop RemembeRL.
[arXiv preprint arXiv:2509.20297 (2025).](https://arxiv.org/abs/2509.20297)
- ``mindmap`` codebase:
- [github.com/nvidia-isaac/nvblox_mindmap](https://github.com/nvidia-isaac/nvblox_mindmap)

## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

## 数据集描述
本数据集是在艾萨克实验室（Isaac Lab）中基于`mindmap`定义的「立方体堆叠（Cube Stacking）」任务生成的多模态轨迹合集。
该任务旨在评估机器人操控策略的空间记忆能力。
依托该（部分）数据集，您可生成用于[mindmap模型](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints/tree/main)训练的完整数据集，亦可开展`mindmap`模型训练或对其开环/闭环性能进行评估。
本数据集仅用于研发用途。

## 数据集所有者
英伟达（NVIDIA）公司

## 数据集创建日期
2025年10月15日

## 许可/使用条款
本数据集受[知识共享署名-非商业性使用4.0国际许可协议（CC BY-NC 4.0）](https://creativecommons.org/licenses/by-nc/4.0/legalcode)约束。

## 适用场景
本数据集的适用场景包括：
1. 基于模仿学习的机器人操控策略研究
2. 在需要空间记忆的任务上训练与评估机器人操控策略
3. 借助[mindmap代码库](https://github.com/nvidia-isaac/nvblox_mindmap)训练并评估[mindmap模型](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints)

## 数据集特征
### 数据采集方法
* 合成数据
* 人类遥操作
* 自动轨迹生成

我们在艾萨克实验室（Isaac Lab）中通过SpaceMouse采集了10段人类遥操作演示数据。基于这10段演示数据，我们通过合成运动轨迹生成框架[Isaac Lab Mimic](https://isaac-sim.github.io/IsaacLab/v2.0.1/source/overview/teleop_imitation.html)自动生成了总计1000段演示数据。

### 标注方法
不适用

## 数据集格式
我们提供了由Mimic生成的1000段演示数据的HDF5格式文件，以及从该HDF5文件转换而来的、适配`mindmap`格式的10段演示数据集。
受存储容量限制，本次仅提供适配`mindmap`格式的10段演示数据。若需生成完整的1000段演示数据，请参阅`mindmap`的[数据生成文档](https://nvidia-isaac.github.io/nvblox_mindmap/pages/data_generation.html)。

`mindmap`格式数据集由打包为tar格式的演示文件夹组成。解压后的目录结构如下：

📂 <数据集名称>/
├── 📂 demo_00000/
│ ├── 00000.<相机名称>_depth.png
│ ├── 00000.<相机名称>_intrinsics.npy
│ ├── 00000.<相机名称>_pose.npy
│ ├── 00000.<相机名称>_rgb.png
│ ├── 00000.nvblox_vertex_features.zst
│ ├── 00000.robot_state.npy
│ ├── ...
│ ├── <演示步数>.<相机名称>_depth.png
│ ├── ...
│ ├── <演示步数>.robot_state.npy
│ └── demo_successful.npy
├── 📂 demo_00001/
│ └── ...
└── 📂 demo_00009/
└── dataset_name.hdf5

每段演示包含可变长度的`NUM_STEPS`步多模态数据：
- RGB-D帧，包含对应相机位姿与内参矩阵
- 以`nvblox_vertex_features.zst`中特征化点云形式存储的度量语义重建结果
- 机器人状态数据，包含末端执行器位姿、夹爪闭合状态与头部偏航角

## 数据集量化
记录数量：1000段演示轨迹
总存储容量：6 GB

## 参考文献
- `mindmap`相关论文：
雷莫·施泰纳（Remo Steiner）、亚历山大·米兰（Alexander Millane）、大卫·廷达勒（David Tingdahl）、克莱门斯·福尔克（Clemens Volk）、维克拉姆·拉马萨米（Vikram Ramasamy）、姚新杰（Xinjie Yao）、彼得·杜（Peter Du）、索哈·普亚（Soha Pouya）及盛诗伟（Shiwei Sheng）。《mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies》，CoRL 2025 研讨会RemembeRL，[arXiv预印本arXiv:2509.20297 (2025)](https://arxiv.org/abs/2509.20297)。
- `mindmap`代码库：[github.com/nvidia-isaac/nvblox_mindmap](https://github.com/nvidia-isaac/nvblox_mindmap)

## 伦理考量
英伟达（NVIDIA）认为，可信人工智能是一项共同责任，我们已制定相关政策与实践规范，以支持各类人工智能应用的开发。开发者在按照服务条款下载或使用本数据集时，应与其内部模型团队协作，确保该模型符合相关行业与应用场景的要求，并防范可能出现的产品误用问题。

请通过[此链接](https://www.nvidia.com/en-us/support/submit-security-vulnerability/)报告安全漏洞或英伟达人工智能相关问题。