HiFi-UMI-2K

# HiFi-UMI-2K: High-Fidelity Robot-Free Manipulation Data <p align="center"> <b>2,000 hours released · 6 synchronized camera views · 480+ scenes · 3 mm pose accuracy · &lt;40 µs synchronization</b> </p> <p align="center"> <a href="https://cloud.simpleai.tech/simple-world-lab/hifi-umi/">🌐 Project Website</a> | <a href="https://huggingface.co/datasets/simple-world-lab/HiFi-UMI-2K">📦 Dataset</a> | <b>📄 Paper: <a href="https://arxiv.org/abs/2607.25895">arXiv:2607.25895</a></b> </p> <p align="center"> <a href="https://cloud.simpleai.tech/simple_world_lab/media/data-canvas.mp4"> <img src="https://cloud.simpleai.tech/simple_world_lab/media/posters/data-canvas.jpg" alt="A mosaic of diverse HiFi-UMI demonstrations" width="960" /> </a> <br/> _{Examples from the HiFi-UMI corpus. Click the image to play the video.} </p> ## 📚 Introduction **HiFi-UMI** is a portable, high-fidelity bimanual capture system for collecting robot-free manipulation demonstrations. **HiFi-UMI-2K** is the large-scale dataset produced by this system and is designed to provide action-grounded supervision for manipulation policy pre-training and post-training without requiring a robot during data collection. The release contains a curated **2,000-hour** subset of a source corpus exceeding **20,000 hours** and **4.32 million episodes** across more than **480 scenes**. Each episode includes synchronized multi-view video, calibrated bimanual end-effector trajectories, gripper states, language annotations, task metadata, and quality-control information. The HiFi-UMI system and dataset accompany the technical report: > **HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone** The work asks whether increasing the fidelity of robot-free data can remove the remaining real-robot teleoperation "anchor" normally used during deployment-oriented post-training. The paper reports that policies trained only on HiFi-UMI task demonstrations can be deployed directly on a real bimanual robot, with no teleoperated robot demonstrations in the training loop. ## 💡 Highlights - **Robot-free collection:** demonstrations are captured directly from natural human manipulation without requiring the target robot, a teleoperation rig, or an instrumented environment. - **High-fidelity trajectories:** head-mounted offline stereo-inertial SLAM and fiducial-marker localization provide approximately **3 mm local end-effector accuracy**. - **Native bimanual pose:** both hands are localized in the same head-camera coordinate frame, providing accurate inter-hand relative pose without post-hoc cross-camera reconstruction. - **Hardware synchronization:** all cameras, IMUs, encoders, and gripper signals share a GPIO hardware trigger, yielding cross-sensor offsets below **40 µs**. - **Ultra-wide six-view sensing:** every episode contains stereo head views and two non-parallel fisheye views per hand, covering approximately **200°** horizontally and vertically around each gripper. - **Replay-validated trajectories:** the processing pipeline achieves approximately **98% trajectory reconstruction success** and **98% WBC replay validation success**. - **Training-ready export:** data is distributed in a LeRobot v3-style format with frame-level Parquet tables, MP4 videos, task text, episode metadata, validity masks, and normalization statistics. - **Deployment-oriented supervision:** the accompanying paper evaluates HiFi-UMI-only post-training on both vision-language-action and world-action-model policy families. ## 📊 Dataset at a Glance | Property | Value | | --- | --- | | Public release | **2,000 hours** | | Source corpus | **20,000+ hours** | | Source-corpus episodes | **4.32M+** | | Collection scenes | **480+** | | Camera views per episode | **6** | | Local end-effector error | **3 mm** | | Cross-sensor time offset | **<40 µs** | | Dropped frames | **<2 per hour** | | Gripper-state error | **<0.1°** | | Trajectory reconstruction success | **98%** | | WBC replay validation success | **98%** | | Storage format | LeRobot v3-style Parquet + MP4 | | License | CC BY 4.0 | The 2,000-hour release is a curated subset of the larger source corpus. Statistics explicitly labeled as source-corpus statistics describe the full processed collection rather than the released subset alone. ## 🧤 HiFi-UMI Capture System The capture hardware is co-designed around four fidelity requirements: 1. **Accurate pose acquisition:** a head-mounted stereo rig and IMU are processed with offline stereo-inertial SLAM. Marker cubes on both hands are localized in the same head-camera frame. 2. **Natural bimanual interaction:** a full-palm, glove-like asymmetric gripper preserves direct contact, natural force distribution, and support for both small and larger objects. 3. **Wide visual coverage:** two head cameras and four hand cameras reduce occlusion and blind spots during contact-rich manipulation. 4. **Online quality control:** the device detects underexposure, motion blur, overly fast motion, and tracking-risk events during collection and provides immediate operator feedback. <p align="center"> <a href="https://cloud.simpleai.tech/simple-world-lab/hifi-umi/"> <img src="https://cloud.simpleai.tech/simple_world_lab/assets/capture-hands-DJ4iVDo1.jpg" alt="HiFi-UMI native bimanual capture hardware with two hands and four hand-camera views" width="400" /> </a> <br/> _{Native bimanual capture hardware: two hands and four hand-camera views.} </p> ## 🔄 High-Fidelity Data Engine Raw captures pass through a closed-loop data-production pipeline: 1. **Collection and upload** with hardware synchronization, online quality monitoring, and operator-provided temporal markers. 2. **Trajectory reconstruction and automatic cleaning** using offline stereo-inertial SLAM and automatic anomaly detection. 3. **WBC replay validation** using whole-body retargeting in simulation to reject kinematically or dynamically infeasible trajectories. 4. **AI-assisted annotation** for task descriptions, subtask boundaries, manipulated objects, abnormal events, and annotation confidence. 5. **Human verification** focused on quality-control flags and low-confidence annotations. 6. **Analysis and export** with task, scene, object, action-pattern, trajectory-quality, and replay-success statistics. Training and evaluation results feed back into later collection plans, allowing the corpus to be rebalanced toward missing tasks, objects, interaction dynamics, and recovery behaviors. <p align="center"> <a href="https://cloud.simpleai.tech/simple_world_lab/media/replays/replay-01.mp4"> <img src="https://cloud.simpleai.tech/simple_world_lab/media/posters/replay-01.jpg" alt="HiFi-UMI trajectory reconstruction with six synchronized camera views" width="760" /> </a> <br/> _{Trajectory reconstruction and six-view replay. Click the image to play the video.} </p> ## ✍️ Trajectory Fidelity The handwriting demonstration below provides a qualitative view of the local trajectory accuracy. The reconstructed bimanual trajectory preserves millimeter-scale pen motion and remains aligned with all six camera streams. <p align="center"> <a href="https://cloud.simpleai.tech/simple_world_lab/media/demos/handwriting.mp4"> <img src="https://cloud.simpleai.tech/simple_world_lab/media/posters/handwriting.jpg" alt="Millimeter-scale handwriting trajectory reconstructed from HiFi-UMI data" width="760" /> </a> <br/> _{Click the image to play the handwriting reconstruction video.} </p> ## 📈 Results Reported in the Paper The paper compares HiFi-UMI-only post-training with conventional in-domain real-robot teleoperation post-training. Architecture, initialization, optimization, action representation, and deployment protocol are held fixed within each model family. ### Zero-robot post-training | Policy family | HiFi-UMI post-training | Teleoperation post-training | Difference | | --- | ---: | ---: | ---: | | StarVLA-QwenPI | 51.3% (82/160) | 53.8% (86/160) | -2.5 points | | OpenPI-π_0.5 | 77.5% (124/160) | 74.4% (119/160) | +3.1 points | | LingBot-VA | 56.9% (91/160) | 57.5% (92/160) | -0.6 points | Additional findings: - Pooling the two VLA families gives **64.4%** (206/320) for HiFi-UMI post-training and **64.1%** (205/320) for teleoperation post-training. - OpenPI-π_0.5 post-trained only on HiFi-UMI reaches **85% success** on Remote Insertion. - The comparisons are evaluated under scene-level distribution shift for the HiFi-UMI condition: UMI demonstrations are collected across different sites, while teleoperation data is collected in the robot evaluation environment. - The comparison measures practical data-production pipelines rather than equal-sample data efficiency: the VLA experiments use approximately 3,200 HiFi-UMI trajectories or 300 teleoperation trajectories per task. ### Large-scale pre-training The paper also pre-trains StarVLA-QwenPI on a 4,000-hour HiFi-UMI mixture: - Held-out action-prediction error decreases by **61%** over one pass through the corpus. - Pre-decay scaling follows a power law with exponent **α = 0.268** and **R² = 0.993**. - Mean action error on ten unseen tasks decreases by **41%**. - HiFi-UMI pre-training raises aggregate real-robot post-training success by **18.1 percentage points** under a controlled initialization comparison. These figures describe the research setting reported in the technical report. They are not universal performance guarantees for every policy, robot embodiment, or downstream task. ## 📂 Repository Layout The repository is sharded at the top level: ```text chunk-XXXX/ └── part-0000/ ├── data/ │ └── chunk-000/ │ └── file-000.parquet ├── videos/ │ ├── observation.images.head_main/ │ │ └── chunk-000/file-000.mp4 │ ├── observation.images.head_main_stereo_right/ │ │ └── chunk-000/file-000.mp4 │ ├── observation.images.left_hand_up/ │ │ └── chunk-000/file-000.mp4 │ ├── observation.images.left_hand_down/ │ │ └── chunk-000/file-000.mp4 │ ├── observation.images.right_hand_up/ │ │ └── chunk-000/file-000.mp4 │ └── observation.images.right_hand_down/ │ └── chunk-000/file-000.mp4 └── meta/ ├── info.json ├── modality.json ├── stats.json ├── tasks.parquet └── episodes/ └── chunk-000/file-000.parquet ``` Path templates are recorded in `meta/info.json`: ```json { "data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet", "video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4" } ``` ## 📊 Frame-Level Data `data/chunk-000/file-000.parquet` is the frame-level training table. Each row corresponds to one synchronized frame. | Column | Type | Description | | --- | --- | --- | | `observation.state` | `float32[20]` | Current bimanual end-effector and gripper state | | `observation.state_valid` | `bool[20]` | Per-dimension validity mask for `observation.state` | | `action` | `float32[20]` | Absolute next-state target action | | `action_valid` | `bool[20]` | Per-dimension validity mask for `action` | | `timestamp` | `float32` | Time relative to the beginning of the episode, in seconds | | `frame_index` | `int64` | Frame number within the episode, starting from 0 | | `episode_index` | `int64` | Episode identifier within the current shard | | `index` | `int64` | Continuous global frame index within the current shard | | `task_index` | `int64` | Index into `meta/tasks.parquet` | | `valid.frame` | `bool` | Whether the frame is recommended for training | Frames with `valid.frame == false` are intentionally retained in both Parquet and video files. This preserves strict alignment between data rows, video frames, and timestamps. Most training pipelines should filter to `valid.frame == true`. ## 🎮 State and Action Representation The state and action vectors contain 20 dimensions: ```text observation.state = [right_10d, left_10d] action = [right_action_10d, left_action_10d] ``` Each hand uses the following 10-dimensional layout: ```text [x, y, z, rot6d_0, rot6d_1, rot6d_2, rot6d_3, rot6d_4, rot6d_5, gripper] ``` | Slice | Meaning | | --- | --- | | `0:3` | End-effector position `xyz`, in meters (`m`) | | `3:9` | 6D rotation representation using the first two rows of a rotation matrix | | `9:10` | Gripper opening angle, in radians (`rad`) | The stored `action` uses the same layout and units and represents an **absolute next-state target**. The final frame normally repeats the final available target so that action rows and video frames remain aligned. The policy implementations in the paper may convert these stored targets into model-specific robot-centric relative actions during training. Users should not assume that the exported `action` is already expressed in the relative action convention used by a particular VLA, WAM, or robot controller. `meta/modality.json` records the semantic slice of every state and action block, including right and left end-effector pose, gripper state, and the mapping between video keys and original camera streams. ## 🧭 Coordinate Frames <p align="center"> <img src="https://cloud.simpleai.tech/simple-world-lab/media/posters/axis_illustration.jpg.png" alt="HiFi-UMI hand, head-camera, and shared world coordinate-frame conventions" width="500" /> <br/> _{Coordinate-frame conventions for the HiFi-UMI capture system.} </p> The dataset uses the following coordinate-frame conventions: | Coordinate frame | Origin | +X axis | +Y axis | +Z axis | | --- | --- | --- | --- | --- | | Right-hand frame | At the fingertip | Forward, along the fingertip pointing direction | To the left | Upward | | Left-hand frame | At the fingertip | Forward, along the fingertip pointing direction | To the left | Upward | | Head-camera frame | At the optical center of the left camera in the head-mounted stereo pair | To the right in the image plane | Downward in the image plane | Forward, along the camera optical axis | Both hand coordinate frames use the same axis convention. The head-camera frame follows the standard optical-camera convention, with `+Z` pointing into the observed scene. The definitions above describe the local coordinate frames attached to the two hands and the head camera. Their time-varying trajectories are all expressed in the same shared world coordinate frame. ### Shared World Coordinate Frame - **Shared frame:** the left-hand, right-hand, and head trajectories of a recording are represented in one common world coordinate system. - **Origin:** the world-frame origin is initialized at an arbitrary position and has no fixed semantic location in the physical scene. - **+Z axis:** the positive `Z` axis is aligned with the direction of gravity. - **XY plane:** the `X` and `Y` axes lie in the plane perpendicular to gravity. Because the world origin is arbitrary, absolute world positions should not be compared directly across different recordings without additional alignment. Within a recording, the head and both hand trajectories are mutually aligned and can be transformed or compared in the shared world frame. ## 🎥 Video Streams Videos are stored as: ```text videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4 ``` Standard video keys: ```text observation.images.head_main observation.images.head_main_stereo_right observation.images.left_hand_up observation.images.left_hand_down observation.images.right_hand_up observation.images.right_hand_down ``` Resolution, FPS, codec, pixel format, and channel count are defined in `meta/info.json["features"]`. Within each video shard, episode frames are concatenated in `episode_index` order. The position of each episode inside every video file is recorded in `meta/episodes/chunk-000/file-000.parquet`: ```text videos/{video_key}/chunk_index videos/{video_key}/file_index videos/{video_key}/from_timestamp videos/{video_key}/to_timestamp ``` To locate the video frame corresponding to a data row: 1. Select the row's `episode_index`. 2. Look up the episode's video file and `from_timestamp`. 3. Add the row's episode-relative `timestamp`. 4. Decode the corresponding frame using the FPS from `meta/info.json`. ## 🏷️ Task Text `meta/tasks.parquet` maps natural-language task descriptions to `task_index`: ```text index.name = "task" columns = ["task_index"] ``` The frame table references this mapping through `data.task_index`. Task strings associated with each episode are also stored in the `tasks` field of the episode metadata. ## 🧾 Episode Metadata `meta/episodes/chunk-000/file-000.parquet` contains one row per episode. | Column | Description | | --- | --- | | `episode_index` | Episode identifier | | `data/chunk_index` | Data Parquet chunk identifier | | `data/file_index` | Data Parquet file identifier | | `dataset_from_index` | Inclusive start index in the frame table | | `dataset_to_index` | Exclusive end index in the frame table | | `tasks` | Task-text list associated with the episode | | `length` | Number of frames in the episode | | `videos/{video_key}/...` | Location of the episode in each video shard | | `stats/{feature}/...` | Episode-level statistics | Expected invariants: ```text length = dataset_to_index - dataset_from_index timestamp = frame_index / fps ``` ## 🗂️ Dataset Metadata `meta/info.json` is the primary entry point: | Field | Description | | --- | --- | | `codebase_version` | Dataset format version | | `robot_type` | Capture device or robot type | | `dataset_id` | Dataset identifier | | `total_episodes` | Number of episodes in the current part | | `total_frames` | Number of frames in the current part | | `total_tasks` | Number of tasks in the current part | | `fps` | Output frame rate | | `splits` | Train-split ranges | | `data_path` | Data Parquet path template | | `video_path` | Video path template | | `features` | Dtypes, shapes, frame rates, and video properties | | `state_layout` | Semantic layout of `observation.state` | | `action_layout` | Semantic layout of `action` | | `video_processing` | Cropping, scaling, encoding, and alignment strategy | | `invalid_policy` | Invalid-frame handling policy | | `source` | Export provenance and media mappings | `meta/stats.json` contains part-level feature statistics: ```text min max mean std count ``` These values can be used for normalization. If a training job uses only `valid.frame == true`, we recommend recomputing normalization statistics after applying the same filtering rule. ## 📖 Citation If you use HiFi-UMI, please cite the technical report: ```bibtex @article{simpleai2026hifiumi, title = {HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone}, author = {{Simple AI} and Wei, Yuteng and Ma, Jinming and Wang, Jiawei and Zhou, Weitao and Zuo, Yushen and Rui, Ke and Li, Minglei and Zhang, Jinhao and Pan, Zhikang and Wang, Xiang and Jia, Haoran and Du, Huan and Zeng, Zicheng and Ma, Jun and Qin, Guiyu and Zhang, Di and Li, Xiaofei}, journal = {arXiv preprint arXiv:2607.25895}, year = {2026} } ``` ## 📜 License The dataset is released under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0). You may share and adapt the data, including for commercial use, provided that appropriate attribution is given, a link to the license is included, and modifications are indicated. ## ❤️ Acknowledgements HiFi-UMI-2K is produced by **Simple AI** with contributions from the capture-system, data-engine, annotation, policy-learning, and real-robot evaluation teams, together with the operators and reviewers who collected and verified the demonstrations. For project updates, paper release information, and additional videos, visit the [HiFi-UMI project website](https://cloud.simpleai.tech/simple-world-lab/hifi-umi/).