so101-vla-datasets

SO-101 VLA数据集是一个用于视觉-语言-动作（VLA）学习的机器人操作数据集集合，使用SO-101（SO-ARM101）从动机器人直接收集桌面操作演示，并以LeRobot v3.0格式整理。数据按对象（如立方体、水果）→任务（如拾取放置）→变体（如基础版本、视觉提示覆盖版本）的层次结构组织子文件夹。数据集包含多种模态：两个RGB摄像头的视频、关节状态和动作数据以及自然语言任务指令。它涵盖了多个子集，例如立方体拾取放置、水果拾取放置和基本操作原语（如拾取、推动、放入），并提供了不同变体以支持VLA和ACT等学习任务。数据集规模在10万到100万样本之间，采用Apache-2.0许可证。SO-101 VLA Dataset is a collection of robotic manipulation datasets for vision-language-action (VLA) learning. It directly collects tabletop manipulation demonstrations using the SO-101 (SO-ARM101) passive robot, and is organized in LeRobot v3.0 format. The dataset organizes its subfolders in a hierarchical structure following the order: object (e.g., cubes, fruits) → task (e.g., pick-and-place) → variant (e.g., base version, visual prompt overlay version). It includes multiple modalities: videos from two RGB cameras, joint state and action data, as well as natural language task instructions. It encompasses multiple subsets, including cube pick-and-place, fruit pick-and-place, and basic manipulation primitives (e.g., pick, push, place), and offers diverse variants to support learning tasks such as VLA and ACT. The dataset contains between 100,000 and 1,000,000 samples, and is licensed under Apache-2.0.

原始信息汇总

数据集概述

• 名称: SO-101 VLA Datasets
• 许可证: Apache-2.0
• 任务类别: 机器人学（robotics）
• 标签: LeRobot, so101, so-arm101, vla, vision-language-action, manipulation, pick-and-place, visual-prompting
• 数据集规模: 100K < n < 1M 帧

机器人平台

• 机器人: SO-ARM101 (so_follower)，6-DoF（6自由度）
• 数据格式: LeRobot v3.0 — data/*.parquet + videos/*.mp4 + meta/
• 模态: RGB摄像头（2个）+ 关节状态/动作 + 自然语言任务指令

数据集结构

数据按 对象(object) → 任务(task) → 变体(variant) 层级组织：

so101-vla-datasets/ ├── cube/pick-place/ # "橙色立方体放入彩色篮子" │ ├── base/ # 原始演示 │ └── vp-sam3-top/ # + SAM3 visual prompt 叠加（俯视图） ├── fruit/pick-place/ # "水果(如香蕉)放入三色篮子" │ ├── base/ # 原始演示 │ ├── vp-sam3-dual/ # + SAM3 visual prompt 叠加（双视图） │ ├── vp-vla/ # + VP-VLA: visual-prompt 事件 & grounding 目标 │ ├── vp-vla-ee7/ # + VP-VLA (ee7 变体) │ ├── act-goal-onehot/ # + goal one-hot 目标 (observation.environment_state) │ └── act-extra/ # ACT 训练的额外演示 ├── primitives/ # 初始任务原语（摄像头/fps 配置不同） │ ├── pickup/ │ ├── push/ │ └── put-inside/ └── eval/ └── groot-green/ # 策略评估用 hold-out

子集详情

子集	对象	Episodes	Frames	FPS	摄像头	说明
cube/pick-place/base	orange cube	74	11,269	10	top, wrist	pick&place 原始
cube/pick-place/vp-sam3-top	orange cube	74	11,269	10	top, wrist	SAM3 VP 叠加 (top)
fruit/pick-place/base	fruit	240	45,349	10	top, wrist	3-basket pick&place 原始
fruit/pick-place/vp-sam3-dual	fruit	240	45,349	10	top, wrist	SAM3 VP 叠加 (dual)
fruit/pick-place/vp-vla	fruit	180	34,382	10	top, wrist	VP 事件 + grounding 目标
fruit/pick-place/vp-vla-ee7	fruit	180	34,382	10	top, wrist	VP-VLA ee7 变体
fruit/pick-place/act-goal-onehot	fruit	180	34,382	10	top, wrist	+ goal one-hot 目标
fruit/pick-place/act-extra	fruit	60	10,967	10	top, wrist	ACT 额外演示
primitives/pickup	—	100	34,618	30	front, top	任务原语
primitives/push	—	50	16,298	30	front, top	任务原语
primitives/put-inside	—	81	25,126	30	front, top	任务原语
eval/groot-green	green	1	389	10	top, wrist	评估 hold-out

pick-place 系列为 10 FPS · top+wrist 摄像头，primitives 系列为 30 FPS · front+top 摄像头，两者采集配置不同。联合训练时需注意摄像头键/帧率匹配。

变体说明

变体	含义
base	未经处理的原始遥操作演示
vp-sam3-*	基于 SAM3 分割的 visual prompt 叠加版本（top = 仅俯视图，dual = 双视图）
vp-vla	VP-VLA 训练用 — meta/vp_events/ 中包含 visual-prompt 事件和 grounding 坐标目标
act-goal-onehot	ACT 训练用 — 表示目标篮子的 goal one-hot 添加到 observation.environment_state

特征 Schema

• action — float32[6]: shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos
• observation.state — float32[6]: 与动作相同的关节位置
• observation.images.{top,wrist} 或 {front,top} — RGB 视频 (mp4)
• 标准 LeRobot 索引: timestamp, frame_index, episode_index, index, task_index
• 任务指令存储在 meta/tasks.parquet 中（例如 "pick the banana and place it in the yellow basket"）

使用方式

通过指定 root= 路径加载特定子集：

python from huggingface_hub import snapshot_download from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

subset = "fruit/pick-place/base" local = snapshot_download( repo_id="kangkb7701/so101-vla-datasets", repo_type="dataset", allow_patterns=f"{subset}/*", ) ds = LeRobotDataset("kangkb7701/so101-vla-datasets", root=f"{local}/{subset}") print(ds, ds.num_episodes, ds.num_frames)

CLI 下载特定子集：

bash hf download kangkb7701/so101-vla-datasets --repo-type dataset --include "cube/pick-place/base/*" --local-dir ./so101_cube_base

引用

bibtex @misc{so101_vla_datasets, title = {SO-101 VLA Datasets}, author = {kangkb7701}, year = {2025}, howpublished = {url{https://huggingface.co/datasets/kangkb7701/so101-vla-datasets}} }