LA-33K

<div align="center"> # LA-33K ### Language-Action Pretraining Data for Vision-Language-Action Models <p align="center"> <a href="https://arxiv.org/abs/2606.27295"><img src="https://img.shields.io/badge/arXiv-2606.27295-b31b1b.svg" alt="arXiv"></a> <a href="https://huggingface.co/papers/2606.27295"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Paper-LA4VLA-blue" alt="Hugging Face Paper"></a> <a href="https://github.com/MINT-SJTU/LA4VLA"><img src="https://img.shields.io/badge/GitHub-LA4VLA-black" alt="GitHub"></a> <a href="https://modelscope.cn/datasets/Dorayakilin/LA-33K"><img src="https://img.shields.io/badge/ModelScope-LA--33K-624AFF" alt="ModelScope Dataset"></a> <a href="#license"><img src="https://img.shields.io/badge/License-MIT-green.svg" alt="License"></a> </p> <p align="center"> <b>33,116</b> episodes &nbsp;|&nbsp; <b>1.52M</b> frames &nbsp;|&nbsp; <b>9,532</b> language tasks &nbsp;|&nbsp; <b>LeRobot v2.1</b> format &nbsp;|&nbsp; <b>Franka</b> robot </p> <p align="center"> <a href="#dataset-summary">Dataset Summary</a> | <a href="#download">Download</a> | <a href="#use-with-la4vla">Use with LA4VLA</a> | <a href="#citation">Citation</a> </p> </div> LA-33K is a Language-Action (LA) pretraining dataset for robot manipulation. It contains 33,116 short, atomic manipulation episodes derived from the LeRobot-format DROID dataset [`cadene/droid_1.0.1`](https://huggingface.co/datasets/cadene/droid_1.0.1) and annotated with fine-grained, vision-agnostic language instructions. The dataset is released with the paper [LA4VLA: Learning to Act without Seeing via Language-Action Pretraining](https://arxiv.org/abs/2606.27295) and the codebase [MINT-SJTU/LA4VLA](https://github.com/MINT-SJTU/LA4VLA). LA-33K exposes language-action supervision that is often implicit in long vision-language-action demonstrations. Each episode pairs a low-level instruction with proprioceptive states and the corresponding action trajectory, making it useful for pretraining policies to learn reusable language-conditioned action priors. ## Highlights - **Language-action supervision:** short atomic instructions aligned with robot action segments. - **Dense temporal grounding:** average episode length is 46.05 frames, compared with 287.83 frames in the original long demonstrations. - **Training-ready format:** LeRobot v2.1-style `meta/`, `data/`, and `videos/` layout. - **LA4VLA integration:** ready for vision-masked LA pretraining with the [LA4VLA](https://github.com/MINT-SJTU/LA4VLA) codebase. ## Dataset Summary | Property | Value | | --------------------- | -------------------------------------------------------------------------- | | ModelScope repository | `Dorayakilin/LA-33K` | | Source dataset | [`cadene/droid_1.0.1`](https://huggingface.co/datasets/cadene/droid_1.0.1) | | Format | LeRobot v2.1-style dataset | | Robot | Franka | | Episodes | 33,116 | | Frames | 1,524,990 | | Language tasks | 9,532 | | Chunks | 34 | | Split | `train` | | FPS | 15 | | Video streams | 3 per episode | | Total videos | 99,348 | | Video resolution | 180 x 320 RGB | | Video codec | AV1 | The three video streams are: - `observation.images.exterior_1_left` - `observation.images.exterior_2_left` - `observation.images.wrist_left` Although videos are included for inspection and compatibility with LeRobot-style tooling, LA pretraining in LA4VLA masks visual inputs and trains from language instructions, robot states, and action trajectories. ## Dataset Structure ```text LA-33K/ |-- meta/ | |-- info.json | |-- tasks.jsonl | |-- episodes.jsonl | `-- episodes_stats.jsonl |-- data/ | |-- chunk-000/ | | `-- episode_000000.parquet | |-- chunk-001/ | `-- ... `-- videos/ |-- chunk-000/ | |-- observation.images.exterior_1_left/ | |-- observation.images.exterior_2_left/ | `-- observation.images.wrist_left/ |-- chunk-001/ `-- ... ``` Per-episode metadata is stored in `meta/episodes.jsonl`. For example: ```json {"episode_index": 0, "tasks": ["Move downward and forward to approach the object while holding nothing"], "length": 70, "subaction": "move"} ``` ## Fields Each parquet episode contains frame-level robot state, action, language, and metadata fields. The main fields are: | Field | Shape / type | Description | | -------------------------------------- | ------------- | ------------------------------------------------------------------- | | `language_instruction` | string | Atomic vision-agnostic instruction used for LA pretraining | | `observation.state` | float32 `[8]` | 7 joint positions plus gripper | | `observation.state.cartesian_position` | float32 `[6]` | End-effector pose: `x, y, z, roll, pitch, yaw` | | `observation.state.joint_position` | float32 `[7]` | Joint positions | | `action` | float32 `[8]` | 7 joint action dimensions plus gripper | | `action.original` | float32 `[7]` | Cartesian action plus gripper: `x, y, z, roll, pitch, yaw, gripper` | | `action.cartesian_position` | float32 `[6]` | Cartesian target/action pose | | `action.cartesian_velocity` | float32 `[6]` | Cartesian velocity | | `camera_extrinsics.*` | float32 `[6]` | Camera extrinsics for each view | | `timestamp` | float32 | Timestamp in seconds | | `frame_index` | int64 | Frame index within the episode | | `episode_index` | int64 | Episode id | | `task_index` | int64 | Language task id | Additional metadata fields include `task_category`, `building`, `collector_id`, `date`, `reward`, `discount`, and `is_episode_successful`. ## Download After release, the dataset can be downloaded from: ```text https://modelscope.cn/datasets/Dorayakilin/LA-33K ``` The repository follows the same LeRobot-style layout as the Hugging Face release. ## Use With LA4VLA The reference training code is available at [MINT-SJTU/LA4VLA](https://github.com/MINT-SJTU/LA4VLA). After downloading LA-33K, set the dataset path in `LA4VLA_1B/dataset/config.yaml`: ```yaml data_groups: franka_eef: la33k: path: /path/to/LA-33K ``` Then compute normalization statistics: ```bash cd LA4VLA/LA4VLA_1B python -m dataset.compute_normstats dataset/config.yaml --action_horizon 50 ``` During LA pretraining, LA4VLA uses `--vision_masked` so the model learns to predict actions from language and state without relying on visual observations. ## Construction Pipeline LA-33K is constructed from existing DROID robot demonstrations provided through the LeRobot-format [`cadene/droid_1.0.1`](https://huggingface.co/datasets/cadene/droid_1.0.1) repository, without additional robot data collection. The LA4VLA pipeline: 1. Extracts temporal cues from robot state trajectories, including static intervals and gripper transitions. 2. Defines an atomic-action vocabulary covering manipulation primitives such as `move`, `grasp`, `lift`, `lower`, `transport`, `place`, `push`, `pull`, `press`, `rotate`, and `reorient`. 3. Uses Qwen-3-VL-Plus as a proposal generator to segment long demonstrations into candidate atomic intervals and produce compact, vision-agnostic instructions. 4. Applies human verification to check temporal boundaries, subaction labels, and language-action alignment. 5. Retains segments with quality score at least 2 out of 3. ## Limitations - The dataset is derived from a subset of DROID demonstrations in [`cadene/droid_1.0.1`](https://huggingface.co/datasets/cadene/droid_1.0.1) and inherits the coverage, embodiment, environment, and collection biases of the source data. - Instructions are generated through a VLM-assisted pipeline and then human-verified; they may still contain residual boundary or wording noise. - LA episodes are short atomic segments and do not by themselves represent complete long-horizon task plans. - Visual streams are included, but the primary LA4VLA pretraining setting masks visual input. ## License LA-33K is released under the MIT License. Copyright (c) 2026 MINT-SJTU. The dataset is derived from DROID demonstrations made available in LeRobot format through [`cadene/droid_1.0.1`](https://huggingface.co/datasets/cadene/droid_1.0.1), so users should also respect the terms and citation requirements of the upstream data. Please cite both LA4VLA and the relevant upstream DROID work when using this dataset. ## Acknowledgements We thank the DROID dataset team for collecting and releasing the original large-scale robot manipulation demonstrations. We also thank the contributors who converted DROID into the LeRobot-format [`cadene/droid_1.0.1`](https://huggingface.co/datasets/cadene/droid_1.0.1) repository, which serves as the source data for LA-33K. ## Citation If you use LA-33K or LA4VLA, please cite: ```bibtex @misc{lin2026la4vlalearningactseeing, title={LA4VLA: Learning to Act without Seeing via Language-Action Pretraining}, author={Tao Lin and Yuxin Du and Yiran Mao and Zewei Ye and Yilei Zhong and Bing Cheng and Yiming Wang and Jiting Liu and Yang Tian and Junchi Yan and Feiran Wu and Zenan Meng and Hu Wei and Yuqian Fu and Gen Li and Bo Zhao}, year={2026}, eprint={2606.27295}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2606.27295} } ``` Paper: https://arxiv.org/abs/2606.27295 Code: https://github.com/MINT-SJTU/LA4VLA