NeoData

# OpenNeoData [](https://creativecommons.org/licenses/by-nc-sa/4.0/) [](https://github.com/huggingface/lerobot) [](https://huggingface.co/datasets/XinzhiEmbodied/OpenNeoData) [](https://modelscope.cn/datasets/NeoteAI/OpenNeoData) [](https://huggingface.co/datasets/XinzhiEmbodied/OpenNeoData) OpenNeoData is a large-scale real-world robot manipulation dataset: **200 k trajectories / 5,002.8 hours** collected on **7 embodiments** — five fixed-arm robot platforms and two handheld UMI device families — with **wrist-mounted visuotactile sensing on every embodiment**. All data is released in [LeRobot](https://github.com/huggingface/lerobot) **v3.0** format, one sub-dataset per embodiment, dual-hosted on Hugging Face and ModelScope. ## Key Features 🔑 - **200 k trajectories** from 7 embodiments, with a total duration of **5,002.8 hours**. - **Tactile-complete**: every gripper carries two gel visuotactile cameras, time-aligned with RGB at 30 fps across the entire dataset. - **Diverse embodiments**: dual-arm ALOHA and ARX-5, single-arm ARX-5, Flexiv Rizon 4s, UR5e/UR7e, and dual-/single-hand UMI devices. - **257 task types** with bilingual EN/CN descriptions, covering contact-rich manipulation, deformable objects, precise placement and bimanual coordination. ## Get started 🔥 ### Download the Dataset Hugging Face ([`XinzhiEmbodied/OpenNeoData`](https://huggingface.co/datasets/XinzhiEmbodied/OpenNeoData)): ```bash # Make sure you have git-lfs installed (https://git-lfs.com) git lfs install git clone https://huggingface.co/datasets/XinzhiEmbodied/OpenNeoData # If you want to clone without large files - just their pointers GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/XinzhiEmbodied/OpenNeoData ``` The dataset is organized as **one folder per embodiment**, so you can download a single robot instead of all 5,000 hours. For example, only `ur`: ```bash pip install -U huggingface_hub hf download XinzhiEmbodied/OpenNeoData --repo-type dataset \ --include "ur/**" --local-dir OpenNeoData ``` or with git sparse-checkout: ```bash git lfs install git init OpenNeoData && cd OpenNeoData git remote add origin https://huggingface.co/datasets/XinzhiEmbodied/OpenNeoData git sparse-checkout init git sparse-checkout set ur git pull origin main ``` ModelScope mirror ([`NeoteAI/OpenNeoData`](https://modelscope.cn/datasets/NeoteAI/OpenNeoData)): ```bash pip install modelscope modelscope download --dataset NeoteAI/OpenNeoData --include "ur/**" --local_dir OpenNeoData ``` ### Load with LeRobot Each embodiment folder is a standalone LeRobot **v3.0** dataset. Our project relies solely on the `lerobot` library — please follow their [installation instructions](https://github.com/huggingface/lerobot). ```python # pip install "lerobot>=0.6" from lerobot.datasets.lerobot_dataset import LeRobotDataset ds = LeRobotDataset("local/openneodata_ur", root="OpenNeoData/ur") item = ds[0] # observation.state / action / observation.images.* ... ``` We would like to express our gratitude to the developers of [lerobot](https://github.com/huggingface/lerobot) for their outstanding contributions to the open-source community. ## Dataset Structure ### Folder hierarchy ``` OpenNeoData ├── aloha # one LeRobot v3.0 dataset per embodiment │ ├── meta │ │ ├── info.json # fps, features, shapes, chunking │ │ ├── tasks.parquet # task_index -> language instruction │ │ ├── episodes/chunk-000/file-000.parquet# per-episode index, lengths, offsets │ │ └── stats.json # per-feature normalization stats │ ├── data │ │ └── chunk-000 │ │ ├── file-000.parquet # states/actions, many episodes per file │ │ └── ... │ └── videos │ ├── observation.images.third_view │ │ └── chunk-000 │ │ ├── file-000.mp4 # ~500 MB, many episodes per file │ │ └── ... │ ├── observation.images.left_wrist_view │ ├── observation.images.left_wrist_left_tactile │ ├── observation.images.left_wrist_right_tactile │ └── ... # per-embodiment key set, see below ├── arx5 ├── arx5_single ├── flexiv ├── ur ├── umi └── umi_single ``` Following LeRobot v3.0, episodes are **aggregated** into ~500 MB parquet/MP4 files (`chunks_size=1000` files per `chunk-XXX` directory); each episode lies entirely within a single file, and `meta/episodes` maps every episode to its file and time offsets. The loader resolves all of this transparently. ### Embodiments and camera streams | folder | robot | arms | video streams per frame | |---|---|---|---| | `aloha` | ALOHA (dual) | 2 | 7 = third + 2 wrist + 4 tactile | | `arx5` | ARX-5 (dual) | 2 | 7 = third + 2 wrist + 4 tactile | | `arx5_single` | ARX-5 (single-arm) | 1 | 4 = third + wrist + 2 tactile | | `flexiv` | Flexiv Rizon 4s (7-DoF) | 1 | 4 = third + wrist + 2 tactile | | `ur` | UR5e / UR7e | 1 | 4 = third + wrist + 2 tactile | | `umi` | UMI (dual-hand) | 2 | 6 = 2 wrist + 4 tactile | | `umi_single` | UMI (single-hand) | 1 | 3 = wrist + 2 tactile | Camera keys follow the pattern `observation.images.third_view`, `observation.images.{left,right}_wrist_view` and `observation.images.{left,right}_wrist_{left,right}_tactile`; `*_tactile` are gel-pad camera streams from the visuotactile sensors. All streams are HEVC (`hvc1`) MP4, 640×360, `yuv420p`, 30 fps, max keyframe interval 10 frames. ### Feature keys | key | fixed-arm robots | UMI | |---|---|---| | `observation.state` | measured joint positions + gripper | raw tracked EEF state | | `action` | commanded joint positions + gripper | raw tracked EEF action | | `observation.eef_pose` | measured EEF pose | — (state is already EEF) | | `action.eef_pose` | commanded EEF pose | — | | `observation.images.*` | 30 fps video streams | 30 fps video streams | Per-embodiment shapes: | folder | `state` / `action` | `eef_pose` | |---|---|---| | `aloha` | 14 (2 × 7) | 20 (2 × 10) | | `arx5` | 14 (2 × 7) | 20 (2 × 10) | | `arx5_single` | 7 | 10 | | `flexiv` | 8 (7 joints + gripper) | 10 | | `ur` | 7 (6 joints + gripper) | 10 | | `umi` | 20 (2 × 10, raw EEF) | — | | `umi_single` | 10 (raw EEF) | — | Each 10-dim EEF block is `[x, y, z, rot6d(6), gripper]` per arm, left arm first. **rot6d** is the first two columns of the rotation matrix flattened column-major: `[R00, R10, R20, R01, R11, R21]`. **UMI note**: the handheld UMI devices have no joint encoders, so `observation.state` / `action` directly carry the raw tracked end-effector trajectory (position + rot6d + gripper per hand); no separate `*.eef_pose` keys are published for `umi` / `umi_single`. ## License and Citation All the data and code within this repo are under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). <!-- Citation: TBD -->