MuseVLA-dataset

<div align="center"> <span style="font-size:32px;">MuseVLA Dataset</span> </div> <p align="center"> <a href="https://arxiv.org/abs/2606.17598"><img src="https://img.shields.io/badge/arXiv-Paper-red?logo=arxiv&logoColor=white" alt="arXiv"></a> <a href="https://github.com/microsoft/MuseVLA"><img src="https://img.shields.io/badge/Code-GitHub-181717?logo=github&logoColor=white" alt="Code Repository"></a> <a href="https://huggingface.co/microsoft/MuseVLA"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue" alt="Hugging Face Model"></a> </p> Multi-modal robot manipulation dataset with synchronized RGB, depth, acoustic, thermal, and radar streams. Released as two parts (`dataset_01/`, `dataset_02/`) sharing the same per-episode layout. Together they cover ~1400 episodes across 11 instructions (towel / clothes / box / item / drink manipulation). ## Per-episode contents ``` {episode_name}/ ├── video.mp4 # RGB, 1280×720, 30 fps ├── mask/video.mp4 # SAM3 segmentation mask ├── depth/images/ # depth jpgs ├── acoustic/{images,images_masked}/ # spectrogram + 3 overlay variants ├── radar/{images,images_masked}/ # radar + 3 overlay variants ├── thermal/{left,right,images,images_masked}/ # left / right / merged / overlays ├── 6d_pose/right_arm.npy # end-effector trajectory ├── arm_command/, arm_status/, ee_command/, hand_status/ # control signals ``` Annotations live under `*_labels/{episode}.json` with per-segment `instruction`, `start`, `end`, plus `_validity.json` recording SAM3 validity. Top-level `instructions_all.json`, `valid_samples.json`, and `statistics_6dpose_delta_NRt0.json` are provided for indexing and normalization. ## Statistics | Episodes | Valid | Annotated segments | Unique instructions | RGB | Frame rate | |---:|---:|---:|---:|---|---| | 1397 (1279 + 118) | 1010 (913 + 97) | 1930 (1737 + 193) | 11 | 1280 × 720 | 30 fps | ## Citation ```bibtex @misc{liu2026musevlaadaptivemultimodalsensing, title={MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation}, author={Xingyuming Liu and Ruichun Ma and Heyu Guo and Qixiu Li and Qingwen Yang and Lin Luo and Shiqi Jiang and Chenren Xu and Jiaolong Yang and Baining Guo}, year={2026}, eprint={2606.17598}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2606.17598}, } ``` ## License CC-BY-NC-4.0 (research use only).