HumanoidArena_raw

# HumanoidArena Raw and Multicam Dataset HumanoidArena is a humanoid manipulation and whole-body control benchmark built around egocentric teleoperation, replay, simulation evaluation, and LeRobot-compatible policy training. This dataset release contains the original HumanoidArena NPZ recordings and replay-generated multicam recordings used by the project. It complements the converted LeRobot training dataset and keeps the low-level controller, human input, robot state, environment state, depth, and video-backed camera streams available for analysis and custom conversion. Project links: - Project page: https://humanoidarena.github.io/ - Paper: https://arxiv.org/abs/2606.17833 - Code: see the HumanoidArena repository release. - LeRobot dataset: https://huggingface.co/datasets/WilliamWang16/HumanoidArena_dataset_v3_1 - Model checkpoints: https://huggingface.co/WilliamWang16/HumanoidArena_models ## Dataset Summary The curated release includes 2,999 NPZ episodes, approximately 627 GB in the source tree. Each episode is stored as one `.npz` file. RGB streams are stored as external `.mp4` files under the sibling `videos/` directory and referenced by relative paths inside the NPZ. The main data flow is: ```text teleoperation -> raw NPZ recording -> replay/rerecord -> multicam NPZ -> LeRobot conversion ``` Most tasks contain four branches: - `sonic/`: raw single-front-camera SONIC recordings. - `twist2/`: raw single-front-camera TWIST2 recordings. - `sonic_multicam_rerecord/`: replay-generated SONIC recordings with multicam video streams. - `twist2_multicam_rerecord/`: replay-generated TWIST2 recordings with multicam video streams. After curation, each regular branch contains 100 episodes, except `HOI_grap_cup/sonic`, which contains 99 raw single-camera episodes after removing an invalid temporary recording. `HOI_football_v2` is the exception: it contains `SONIC/` and `TWIST2/` raw recordings only in the current source layout. ## Directory Layout ```text HumanoidArena/ HOI_grap_cup/ sonic/<operator>/*.npz sonic/<operator>/videos/*_front_rgb.mp4 twist2/<operator>/*.npz twist2/<operator>/videos/*_front_rgb.mp4 sonic_multicam_rerecord/<operator>/*.npz sonic_multicam_rerecord/<operator>/videos/*_{front,left_wrist,right_wrist}_rgb.mp4 twist2_multicam_rerecord/<operator>/*.npz twist2_multicam_rerecord/<operator>/videos/*_{front,left_wrist,right_wrist}_rgb.mp4 ... HOI_football_v2/ SONIC/<operator>/*.npz TWIST2/<operator>/*.npz ``` File names follow the simulator task name and a microsecond timestamp: ```text <task_name>_sonic_<timestamp_us>.npz <task_name>_<timestamp_us>.npz ``` ## Tasks and Counts | Task | Branch | NPZ episodes | MP4 videos | Operators | | --- | --- | ---: | ---: | --- | | `HOI_double_desk` | `sonic` | 100 | 100 | `tw` | | `HOI_double_desk` | `sonic_multicam_rerecord` | 100 | 300 | `tw` | | `HOI_double_desk` | `twist2` | 100 | 100 | `zz` | | `HOI_double_desk` | `twist2_multicam_rerecord` | 100 | 300 | `zz` | | `HOI_football_v2` | `SONIC` | 100 | 0 | `tw`, `zk` | | `HOI_football_v2` | `TWIST2` | 100 | 0 | `tw`, `yb` | | `HOI_grap_cup` | `sonic` | 99 | 99 | `yb` | | `HOI_grap_cup` | `sonic_multicam_rerecord` | 100 | 300 | `yb` | | `HOI_grap_cup` | `twist2` | 100 | 100 | `zz` | | `HOI_grap_cup` | `twist2_multicam_rerecord` | 100 | 300 | `zz` | | `HOI_pp_box` | `sonic` | 100 | 100 | `yb` | | `HOI_pp_box` | `sonic_multicam_rerecord` | 100 | 300 | `yb` | | `HOI_pp_box` | `twist2` | 100 | 100 | `yb` | | `HOI_pp_box` | `twist2_multicam_rerecord` | 100 | 300 | `yb` | | `HSI_boxing` | `sonic` | 100 | 100 | `zz` | | `HSI_boxing` | `sonic_multicam_rerecord` | 100 | 300 | `zz` | | `HSI_boxing` | `twist2` | 100 | 100 | `zz` | | `HSI_boxing` | `twist2_multicam_rerecord` | 100 | 300 | `zz` | | `HSI_open_door` | `sonic` | 100 | 100 | `zz` | | `HSI_open_door` | `sonic_multicam_rerecord` | 100 | 300 | `zz` | | `HSI_open_door` | `twist2` | 100 | 100 | `zz` | | `HSI_open_door` | `twist2_multicam_rerecord` | 100 | 300 | `zz` | | `HSI_sit_sofa` | `sonic` | 100 | 100 | `zz` | | `HSI_sit_sofa` | `sonic_multicam_rerecord` | 100 | 300 | `zz` | | `HSI_sit_sofa` | `twist2` | 100 | 100 | `zz` | | `HSI_sit_sofa` | `twist2_multicam_rerecord` | 100 | 300 | `zz` | | `HSI_vision_navi` | `sonic` | 100 | 100 | `yb` | | `HSI_vision_navi` | `sonic_multicam_rerecord` | 100 | 300 | `yb` | | `HSI_vision_navi` | `twist2` | 100 | 100 | `yb` | | `HSI_vision_navi` | `twist2_multicam_rerecord` | 100 | 300 | `yb` | Approximate task sizes in the source tree: | Task | Size | | --- | ---: | | `HOI_double_desk` | 71 GB | | `HOI_football_v2` | 201 GB | | `HOI_grap_cup` | 104 GB | | `HOI_pp_box` | 54 GB | | `HSI_boxing` | 37 GB | | `HSI_open_door` | 51 GB | | `HSI_sit_sofa` | 49 GB | | `HSI_vision_navi` | 63 GB | ## NPZ Schemas Each NPZ contains scalar metadata plus frame-major arrays. `T` denotes the number of frames in the episode. ### Common Metadata and Environment Fields Common fields include: - `schema_version`: schema identifier. - `task`: Isaac Lab task name. - `num_frames`: number of frames. - `episode_object_seed`, `episode_object_seed_source`: scene randomization provenance when available. - `robot_qpos_before_decimation`: `(T, 29)` Unitree G1 joint positions. - `robot_qvel_before_decimation`: `(T, 29)` Unitree G1 joint velocities. - `robot_root_position`: `(T, 3)`. - `robot_root_orientation`: `(T, 4)` in `wxyz` order. - `vla_state`: `(T, 64)` canonical VLA observation state. - `vla_action`: `(T, 40)` canonical VLA reference-pose action. Environment object state is stored with names such as: ```text env_obj_<name>_position env_obj_<name>_orientation env_obj_<name>_linear_velocity env_obj_<name>_angular_velocity episode_init_env_obj_<name>_position episode_init_env_obj_<name>_orientation ``` ### Vision Fields Current episodes use video-backed RGB storage: - `vision_storage_format = "video_v1"` - `vision_rgb_video_path`: relative path to the front RGB MP4. - `vision_rgb_video_fps` - `vision_rgb_video_num_frames` - `vision_frame_indices` - `vision_depth`: `(T, 480, 640, 1)` float16 depth array when present. Multicam rerecord episodes additionally include wrist camera streams: - `vision_left_wrist_rgb_video_path` - `vision_left_wrist_frame_indices` - `vision_left_wrist_depth` - `vision_right_wrist_rgb_video_path` - `vision_right_wrist_frame_indices` - `vision_right_wrist_depth` Some tools also support optional world/perspective camera fields such as `vision_world_rgb_video_path`. ### SONIC Episodes Raw SONIC recordings use: ```text schema_version = sonic_episode_v3 ``` Replay-generated multicam SONIC recordings use: ```text schema_version = sonic_episode_v4_multicam ``` Important SONIC-specific fields include: - Human input and pose: `human_left_hand`, `human_right_hand`, `human_smpl_joints`, `human_smpl_pose`, `human_body_pos`, `human_body_quat_w`, `human_joint_pos`. - Frame markers: `frame_index`, `raw_frame_index`, `consumed_frame_index`, `episode_step`, timestamp fields. - Encoder fields: `encoder_input` `(T, 1762)`, `encoder_latent` `(T, 64)`, and encoder history windows. - Decoder fields: `decoder_obs` `(T, 994)`, `decoder_raw_action` `(T, 29)`, `decoder_target_action` `(T, 29)`. - Executed actions: `final_body_action_29dof`, `hand_action_left`, `hand_action_right`. - VLA fields: `vla_action_raw`, `vla_action_executed`, `vla_action_heading_aligned`, and the common `vla_state` / `vla_action`. Example observed raw SONIC episode: ```text schema_version: sonic_episode_v3 num_frames: 1214 vision_rgb_video_path: videos/..._front_rgb.mp4 vision_depth: (1214, 480, 640, 1) float16 vla_state: (1214, 64) float32 vla_action: (1214, 40) float32 encoder_input: (1214, 1762) float32 decoder_obs: (1214, 994) float32 ``` ### TWIST2 Episodes Raw TWIST2 recordings use: ```text schema_version = twist2_episode_v2 ``` Replay-generated multicam TWIST2 recordings use: ```text schema_version = twist2_episode_v3_multicam ``` Important TWIST2-specific fields include: - Human input: `human_hand_left`, `human_hand_right`, `human_neck`, controller binary trigger fields. - TWIST2 policy inputs and state: `robot_twist2_inference_qpos`, `robot_obs_buf`. - Native TWIST2 command: `robot_action_mimic` `(T, 35)`. - VLA fields: `vla_state`, `vla_state_root_rot6d`, `vla_action`, `vla_action_root_xy_delta`, `vla_action_root_z`, `vla_action_root_rot6d`, `vla_action_joint_pos_29`, `vla_action_hand_binary`. Example observed raw TWIST2 episode: ```text schema_version: twist2_episode_v2 num_frames: 746 vision_rgb_video_path: videos/..._front_rgb.mp4 vision_depth: (746, 480, 640, 1) float16 vla_state: (746, 64) float32 vla_action: (746, 40) float32 robot_action_mimic: (746, 35) float32 ``` ## VLA State and Action Layout The converted LeRobot V3.1 protocol uses: ```text schema: unitree_g1_gmt_refpose_v3_1 robot_type: unitree_g1_refpose_v3_1 state_dim: 64 action_dim: 40 ``` `vla_state` layout: ```text state[0:6] root heading-canonical rot6d state[6:35] canonical Unitree G1 29-DoF joint position state[35:64] canonical Unitree G1 29-DoF joint velocity ``` `vla_action` layout: ```text action[0:2] root reference local XY delta action[2] root Z action[3:9] root reference rot6d action[9:38] canonical Unitree G1 29-DoF joint target action[38:40] left/right binary hand command ``` The VLA action is a reference motion target. It is not a residual action relative to the current simulated robot pose. ## Official Conversion The official conversion entry point is: ```bash cd <repo_root>/isaaclab_twist2_g1 bash tools/data_tools/batch_convert_rotlocal_v3.sh \ --input-root <raw_dataset_root>/HumanoidArena \ --output-root <lerobot_output_root> ``` This script batch-converts first-level `HOI_*` and `HSI_*` IsaacLab recordings from the default `sonic` and `twist2` backend directories into LeRobot datasets using the refpose V3.1 protocol: ```text schema: unitree_g1_gmt_refpose_v3_1 robot_type: unitree_g1_refpose_v3_1 state_dim: 64 action_dim: 40 ``` Useful options: - `--tasks A,B`: convert only selected task directories or canonical task names. - `--overwrite`: replace existing output dataset directories. - `--limit N`: convert only the first `N` episodes per selected backend. - `--use-images`: store images in parquet instead of LeRobot videos. - `--allow-missing-video-frames`: pass missing-frame tolerance through to the converters. - `--skip-convert`: run only post-processing steps. - `--skip-instructions`: skip writing language instructions. - `--skip-gripper-fix`: skip the hand-binary gripper statistics patch. - `--dry-run`: print commands without executing them. By default, the script also writes the task language instructions and patches `meta/stats.json` for `action.hand_binary.left` and `action.hand_binary.right` so binary hand commands use stable fixed statistics. ## Loading Example ```python from pathlib import Path import cv2 import numpy as np episode_path = Path("HumanoidArena/HOI_grap_cup/sonic/yb/example.npz") with np.load(episode_path, allow_pickle=True) as data: print("schema:", data["schema_version"].item()) print("task:", data["task"].item()) print("frames:", int(data["num_frames"])) print("state:", data["vla_state"].shape) print("action:", data["vla_action"].shape) video_path = episode_path.parent / data["vision_rgb_video_path"].item() cap = cv2.VideoCapture(str(video_path)) ok, frame_bgr = cap.read() cap.release() if ok: frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB) print("first RGB frame:", frame_rgb.shape) ``` ## Intended Uses This release is intended for: - Replaying HumanoidArena teleoperation trajectories. - Inspecting human input, controller internals, robot state, and environment state. - Reconstructing or validating LeRobot-compatible VLA datasets. - Training and evaluating policies that need raw controller or multicam observations. For standard policy training, use the converted LeRobot dataset unless you specifically need raw NPZ fields. ## Limitations - The dataset is collected in simulation and should not be treated as direct real-robot sensor data. - `HOI_football_v2` has a different release layout and does not include multicam rerecord branches in the inspected source tree. - The curated tree removes temporary, legacy, and orphaned recordings from the regular raw/multicam branches. `HOI_grap_cup/sonic` has 99 raw single-camera episodes because one invalid temporary recording was removed. - Users should still check `vision_storage_format` and `vision_*_video_path` before assuming camera availability in downstream tooling. - Large simulation assets, model checkpoints, and code are released separately from this raw dataset. ## Citation If you use HumanoidArena, please cite: ```bibtex @article{wang2026humanoidarena, title={HumanoidArena: Benchmarking Egocentric Hierarchical Whole-body Learning}, author={Wang, Taowen and Xie, Zikang and Yang, Bin and others}, journal={arXiv preprint arXiv:2606.17833}, year={2026} } ``` ## License and Terms HumanoidArena builds on multiple open-source projects and third-party assets, including TWIST2, SONIC / GR00T Whole-Body Control, LeRobot, Unitree Sim IsaacLab, and ArtVIP assets. Please review the code repository license files, upstream licenses, model/data artifact terms, and simulator asset terms before redistribution or commercial use.