gsethia08/part4-roll_ball

--- license: apache-2.0 task_categories: - robotics tags: - LeRobot configs: - config_name: default data_files: data/*/*.parquet --- This dataset was created using [LeRobot](https://github.com/huggingface/lerobot). <a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=gsethia08/part4-roll_ball"> <img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/> <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/> </a> ## Dataset Description - **Homepage:** [More Information Needed] - **Paper:** [More Information Needed] - **License:** apache-2.0 ## Dataset Structure [meta/info.json](meta/info.json): ```json { "codebase_version": "v3.0", "robot_type": "hand", "total_episodes": 1286, "total_frames": 378085, "total_tasks": 5, "chunks_size": 1000, "data_files_size_in_mb": 100, "video_files_size_in_mb": 200, "fps": 30, "splits": { "train": "0:1286" }, "data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet", "video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4", "features": { "observation.camera.ego": { "dtype": "video", "shape": [ 1080, 1920, 3 ], "names": [ "height", "width", "rgb" ], "info": { "video.height": 1080, "video.width": 1920, "video.codec": "av1", "video.pix_fmt": "yuv420p", "video.is_depth_map": false, "video.fps": 30, "video.channels": 3, "has_audio": false } }, "observation.state": { "dtype": "float32", "shape": [ 48 ], "names": { "axes": [ "left_wrist_x", "left_wrist_y", "left_wrist_z", "left_rot_0", "left_rot_1", "left_rot_2", "left_rot_3", "left_rot_4", "left_rot_5", "left_thumb_x", "left_thumb_y", "left_thumb_z", "left_index_x", "left_index_y", "left_index_z", "left_middle_x", "left_middle_y", "left_middle_z", "left_ring_x", "left_ring_y", "left_ring_z", "left_little_x", "left_little_y", "left_little_z", "right_wrist_x", "right_wrist_y", "right_wrist_z", "right_rot_0", "right_rot_1", "right_rot_2", "right_rot_3", "right_rot_4", "right_rot_5", "right_thumb_x", "right_thumb_y", "right_thumb_z", "right_index_x", "right_index_y", "right_index_z", "right_middle_x", "right_middle_y", "right_middle_z", "right_ring_x", "right_ring_y", "right_ring_z", "right_little_x", "right_little_y", "right_little_z" ] } }, "camera.extrinsics": { "dtype": "float32", "shape": [ 16 ], "names": { "axes": [ "extrinsic_0", "extrinsic_1", "extrinsic_2", "extrinsic_3", "extrinsic_4", "extrinsic_5", "extrinsic_6", "extrinsic_7", "extrinsic_8", "extrinsic_9", "extrinsic_10", "extrinsic_11", "extrinsic_12", "extrinsic_13", "extrinsic_14", "extrinsic_15" ] } }, "camera.intrinsics": { "dtype": "float32", "shape": [ 9 ], "names": { "axes": [ "intrinsic_0", "intrinsic_1", "intrinsic_2", "intrinsic_3", "intrinsic_4", "intrinsic_5", "intrinsic_6", "intrinsic_7", "intrinsic_8" ] } }, "action": { "dtype": "float32", "shape": [ 48 ], "names": { "axes": [ "action_0", "action_1", "action_2", "action_3", "action_4", "action_5", "action_6", "action_7", "action_8", "action_9", "action_10", "action_11", "action_12", "action_13", "action_14", "action_15", "action_16", "action_17", "action_18", "action_19", "action_20", "action_21", "action_22", "action_23", "action_24", "action_25", "action_26", "action_27", "action_28", "action_29", "action_30", "action_31", "action_32", "action_33", "action_34", "action_35", "action_36", "action_37", "action_38", "action_39", "action_40", "action_41", "action_42", "action_43", "action_44", "action_45", "action_46", "action_47" ] } }, "language_instruction": { "dtype": "string", "shape": [ 1 ], "names": null }, "timestamp": { "dtype": "float32", "shape": [ 1 ], "names": null }, "frame_index": { "dtype": "int64", "shape": [ 1 ], "names": null }, "episode_index": { "dtype": "int64", "shape": [ 1 ], "names": null }, "index": { "dtype": "int64", "shape": [ 1 ], "names": null }, "task_index": { "dtype": "int64", "shape": [ 1 ], "names": null } } } ``` ## Citation **BibTeX:** ```bibtex [More Information Needed] ```This dataset is a robotics dataset focused on hand robot manipulation. It was created using LeRobot and consists of 1286 episodes with a total of 378085 frames, covering 5 distinct tasks. The dataset provides video data at 30fps with a resolution of 1080x1920, along with detailed robot state information including positions and rotations of the left and right wrists, as well as coordinates for each finger (thumb, index, middle, ring, little). Additionally, it includes camera extrinsic and intrinsic parameters, action data, language instructions, and metadata such as timestamps, frame indices, and episode indices. The data is stored in Parquet format and is suitable for research in robotics, imitation learning, or reinforcement learning.

搜集汇总

数据集介绍

构建方式

该数据集依托LeRobot框架构建，采用统一的机器人数据采集与存储规范。数据以Parquet列式格式分块存储，视频流则编码为AV1格式的MP4文件，兼顾存储效率与读取性能。全部1286个回合共378085帧，以30帧每秒的频率记录，覆盖5项不同的操作任务。每帧同步采集手部本体感觉状态与动作向量，各含48维浮点数据，并附带第一人称视角的高清视频、相机内外参、语言指令及时间戳等元信息，形成了多模态、时间对齐的完整数据单元。

特点

数据集面向手部机器人操作场景，核心特征体现在多模态同步与细粒度状态表达上。观测状态与动作向量均细分为左右手腕位置、旋转及五指关节坐标，共48维，精确刻画双手的复杂运动。第一人称视频分辨率达1920×1080，配合相机内参和外参，为视觉-运动关联研究提供空间参照。数据集包含1286个回合和5项任务，规模适中，既保证任务多样性，又便于模型训练与评估。语言指令字段为任务理解与多模态学习提供了自然语言监督信号。

使用方法

使用该数据集时，可通过LeRobot官方工具库加载Parquet数据文件与MP4视频流，按data_path和video_path模板解析分块路径。数据已预设训练集划分，覆盖全部1286个回合。研究者可利用observation.state与action字段进行模仿学习或策略训练，结合observation.camera.ego视频流开展视觉表征学习，借助language_instruction实现语言条件化控制。相机内外参可用于三维空间重建或视角变换。数据集亦支持通过Hugging Face Spaces的在线可视化工具进行交互式浏览与质量核验。

背景与挑战

背景概述

在具身智能与机器人学习迅猛演进的浪潮中，高质量、大规模的操作数据集成为驱动灵巧手策略泛化的关键基础设施。part4-roll_ball数据集依托LeRobot框架构建，于2024年前后发布，由Hugging Face社区贡献者gsethia08创建，聚焦于手部机器人完成滚动球体等精细操作任务。该数据集包含1286条轨迹、378085帧、5项任务，以30fps同步采集1920×1080第一人称视觉、48维手部关节状态与动作、相机内外参及语言指令，为视觉-语言-动作模型提供了多模态对齐的真实世界示范，其规模与丰富度在手部操作领域具有显著的推动意义。

当前挑战

该数据集所应对的核心领域难题在于，灵巧手在非结构化环境中执行滚动球体等接触密集型任务时，需同时处理高维关节协调、复杂接触动力学与视觉遮挡，策略学习极易因状态分布偏移而失效；同时，构建过程亦面临多模态数据时空严格对齐、双手48自由度动作空间精确标注、长时程任务中语言指令与视觉观测的语义一致性保障等工程挑战。此外，1286条轨迹虽初具规模，但任务多样性仅覆盖5类，跨物体、跨场景泛化能力仍受限于示范分布，如何借助该数据集提升策略在未知球体属性与扰动下的鲁棒性，构成后续研究的重点方向。

常用场景

经典使用场景

在机器人学习领域，尤其是灵巧操作与模仿学习方向，part4-roll_ball数据集扮演着关键的角色。其经典使用场景聚焦于基于视觉的双手灵巧操作策略学习，具体任务为“滚动球体”。数据集通过第一人称视角摄像头捕获高分辨率视觉观测，并同步记录双手共48维的关节状态与动作序列，为模仿学习算法提供了丰富的示范数据。研究者通常将其用于训练端到端的策略网络，如行为克隆或扩散策略，使机器人能够从人类示范中学习复杂的双手协调动作。该数据集在LeRobot框架下构建，天然支持大规模机器人学习实验，是推动灵巧手操作研究的重要基准。

解决学术问题

该数据集有效解决了机器人模仿学习中双手灵巧操作示范数据稀缺的难题。在学术研究中，双手协调、高维动作空间以及视觉-运动映射的复杂性长期制约着策略泛化能力的提升。part4-roll_ball通过提供1286个回合、378085帧的多模态数据，涵盖了视觉观测、关节状态、动作指令及语言指令，为研究高维动作空间下的策略学习、跨模态对齐以及任务泛化提供了坚实的实证基础。其意义在于，推动了模仿学习算法在真实双手操作场景中的可重复性研究，并为评估算法在复杂任务上的表现树立了标准化基准，对机器人学习社区产生了深远影响。

衍生相关工作

基于part4-roll_ball数据集，研究者已衍生出一系列经典工作。在模仿学习领域，该数据集常被用于验证行为克隆、扩散策略及隐式行为克隆等算法的有效性，并催生了针对高维动作空间的正则化与数据增强方法。在机器人基础模型方面，该数据集作为LeRobot生态的一部分，促进了多任务、多本体策略的预训练与微调研究。此外，其语言指令标注启发了视觉-语言-动作模型的发展，推动了语言引导的灵巧操作研究。这些衍生工作不仅提升了双手操作的性能上限，也为后续数据集构建与算法评估提供了重要参考。