ningweicn/ShareRobot

---
# For reference on dataset card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/datasets-cards
{}
---

# ShareRobot Dataset

**ShareRobot**, a high-quality heterogeneous dataset that labels multi-dimensional information, including task planning, object affordance, and end-effector trajectory, effectively enhancing various robotic capabilities.

## Overview of ShareRobot Dataset
For **planning**, we have 51,403 episodes and each with 30 frames. In the process of data generation, we design 5 different templates for each of the 10 question types in RoboVQA [1]. In the process of data generation, we randomly select 2 templates of each question type to generate question-answer pairs for every instance. This process transforms 51,403 instances into 1,027,990 question-answer pairs, with annotators monitoring data generation to maintain the dataset’s integrity.

For **Affordance**, we have 6,522 images and each with affordance areas aligned with an instruction.

For **Trajectory**, we have 6,870 images and each with at least 3 {x, y} coordinates aligned with an instruction.

## Dataset Sources


**ShareRobot** dataset contains 23 original datasets from Open X-Embodiment dataset [2], 12 embodiments and 107 types of atomic tasks.

### Raw Dataset for Planning

| Raw Dataset | Number of Raws |
|:-------------------------------------------------------------:| --------------:|
| nyu_door_opening_surprising_effectiveness | 421 |
| bridge | 15738 |
| dlr_edan_shared_control_converted_externally_to_rlds | 63 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 92 |
| cmu_stretch | 10 |
| asu_table_top_converted_externally_to_rlds | 109 |
| dlr_sara_pour_converted_externally_to_rlds | 51 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 27 |
| robo_set | 18164 |
| dobbe | 5200 |
| berkeley_autolab_ur5 | 882 |
| qut_dexterous_manpulation | 192 |
| aloha_mobile | 264 |
| dlr_sara_grid_clamp_converted_externally_to_rlds | 40 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 569 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 39 |
| jaco_play | 956 |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 64 |
| conq_hose_manipulation | 56 |
| fmb | 7836 |
| plex_robosuite | 398 |
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 189 |
| viola | 44 |

### Raw Dataset for Affordance

| Raw Dataset | Number of Raws |
|:-------------------------------------------------------------:| -------------:|
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 24 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 23 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 10 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 112 |
| nyu_door_opening_surprising_effectiveness | 85 |
| jaco_play | 171 |
| bridge | 2610 |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 12 |
| asu_table_top_converted_externally_to_rlds | 24 |
| viola | 1 |
| berkeley_autolab_ur5 | 122 |
| aloha_mobile | 23 |
| conq_hose_manipulation | 1 |
| dobbe | 717 |
| fmb | 561 |
| plex_robosuite | 13 |
| qut_dexterous_manpulation | 16 |
| robo_set | 1979 |
| dlr_edan_shared_control_converted_externally_to_rlds | 18 |
| **Summary** | 6522 |

### Raw Dataset for Trajectory

| Raw Dataset | Number of Raws |
|:-------------------------------------------------------------:| -------------:|
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 35 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 36 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 19 |
| dlr_sara_grid_clamp_converted_externally_to_rlds | 1 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 109 |
| nyu_door_opening_surprising_effectiveness | 74 |
| jaco_play | 175 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 7 |
| bridge | 2986 |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 12 |
| asu_table_top_converted_externally_to_rlds | 22 |
| berkeley_autolab_ur5 | 164 |
| dobbe | 759 |
| fmb | 48 |
| qut_dexterous_manpulation | 29 |
| robo_set | 2374 |
| dlr_sara_pour_converted_externally_to_rlds | 3 |
| dlr_edan_shared_control_converted_externally_to_rlds | 17 |
| **Summary** | 6870 |

## Data Format

### Planning


```json
{
"id"{
"id": 0,
"task": "Future_Prediction_Task",
"selected_step": 3,
"conversations": [
{
"from": "human",
"value": "<image 0-25> After <move the grasped banana towards the mug>, what's the most probable next event?"
},
{
"from": "gpt",
"value": "<place the banana into the mug>"
}
],
"image": [
"/path/to/image_0-25"
]
}
}
```

### Affordance

<!---->
<div style="display: flex; gap: 10px;">
<img src="./images/2d94d985-d47e-4899-9760-c1cb8f19cd89.png" style="width: 300px;" />
<img src="./images/a7817c0b-04b1-4a7c-9535-f9ff7801a689.png" style="width: 300px;" />
</div>

```json
{

"id": 2486,
"meta_data": {
"original_dataset": "bridge",
"original_width": 640,
"original_height": 480
},
"instruction": "place the red fork to the left of the left burner",
"affordance": {
"x": 352.87425387858815,
"y": 186.47871614766484,
"width": 19.296008229513156,
"height": 14.472006172134865
}
```

#### Visualize Code

```python
import json
import os
import cv2
import numpy as np

img_dir = '/path/to/your/original/images/dir'
affordance_json = '/path/to/your/affordances/json'
output_img_dir = '/path/to/your/visualized/images/dir'

with open(affordance_json, 'r') as f:
data = json.load(f)
for item in data:
filepath = os.path.join(img_dir, item['id'])

image = cv2.imread(filepath)
color = (255, 0, 0)
thickness = 2

x_min,y_min = item['affordance']['x'], item['affordance']['y']
x_max,y_max = item['affordance']['x']+item['affordance']['width'], item['affordance']['y']+item['affordance']['height']

# 定义矩形的四个顶点坐标
pts = np.array([
[x_min, y_min], # 左上角
[x_max, y_min], # 右上角
[x_max, y_max], # 右下角
[x_min, y_max] # 左下角
], dtype=np.float32)

# 绘制矩形框
cv2.polylines(image, [pts.astype(int)], isClosed=True, color=color, thickness=thickness)

# 获取相对路径并拼接目标路径
relative_path = os.path.relpath(filepath, img_dir) # 获取相对于 img_dir 的相对路径
output_img_path = os.path.join(output_img_dir, relative_path) # 拼接目标路径

# 创建目标文件夹
output_directory = os.path.dirname(output_img_path)
if not os.path.exists(output_directory):
os.makedirs(output_directory)

# 打印调试信息
print(f"Input filepath: {filepath}")
print(f"Output image path: {output_img_path}")
print(f"Output directory: {output_directory}")

# 保存图像
cv2.imwrite(output_img_path, image)

```

### Trajectory

<!-- -->
<div style="display: flex; gap: 10px;">
<img src="./images/5b923b31-dbbf-470f-af09-5125f5b91ab0.png" style="width: 300px;" />
<img src="./images/1af4535a-acc3-4417-ae33-675f4301f560.png" style="width: 300px;" />
</div>

```json
{
"id": 456,
"meta_data": {
"original_dataset": "bridge",
"original_width": 640,
"original_height": 480
},
"instruction": "reach for the carrot",
"points": [
[
265.45454545454544,
120.0
],
[
275.1515151515152,
162.42424242424244
],
[
280.0,
213.33333333333331
],
[
280.0,
259.3939393939394
]
]
},
```

#### Visualize Code

```python
import json
import os
from PIL import Image, ImageDraw

trajectory_final = '/path/to/your/trajectory_json'
img_dir = '/path/to/your/original/images/dir'
output_img_dir = '/path/to/your/visualzed/images/dir'

with open(trajectory_final, 'r') as f:
data = json.load(f)
for item in data:
filepath = os.path.join(img_dir, item['id'])
points = item['points']

image = Image.open(filepath).convert("RGB") # 确保图像是 RGB 模式
draw = ImageDraw.Draw(image) # 创建绘图对象
# 定义颜色和线宽
color = (255, 0, 0) # 红色 (RGB 格式)
thickness = 2

scaled_points = [
(point[0], point[1])
for point in points
]
# 按照顺序连接相邻的点
for i in range(len(scaled_points) - 1):
draw.line([scaled_points[i], scaled_points[i + 1]], fill=color, width=thickness)

# 获取相对路径并拼接目标路径
relative_path = os.path.relpath(filepath, img_dir)
output_img_path = os.path.join(output_img_dir, relative_path)

# 创建目标文件夹
output_directory = os.path.dirname(output_img_path)
if not os.path.exists(output_directory):
os.makedirs(output_directory)

# 打印调试信息
print(f"Input filepath: {filepath}")
print(f"Output image path: {output_img_path}")
print(f"Output directory: {output_directory}")

# 保存图像
image.save(output_img_path)
```

## Evaluation

## Reference

[1] Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan,Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi,et al. Robovqa: Multimodal long-horizon reasoning forrobotics. In ICRA, pages 645–652, 2024.

[2] Abby O’Neill, Abdul Rehman, Abhinav Gupta, AbhiramMaddukuri, Abhishek Gupta, Abhishek Padalkar, AbrahamLee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al.Open x-embodiment: Robotic learning datasets and rt-xmodels. arXiv preprint arXiv:2310.08864, 2023.

## Citation
```
@article{ji2025robobrain,
title={RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete},
author={Ji, Yuheng and Tan, Huajie and Shi, Jiayu and Hao, Xiaoshuai and Zhang, Yuan and Zhang, Hengyuan and Wang, Pengwei and Zhao, Mengdi and Mu, Yao and An, Pengju and others},
journal={arXiv preprint arXiv:2502.21257},
year={2025}
}
```

# ShareRobot 数据集

**ShareRobot 数据集**是一款高质量异构数据集，标注了任务规划（task planning）、物体功能属性（object affordance）以及末端执行器轨迹（end-effector trajectory）等多维度信息，可有效提升机器人的多项核心能力。

## ShareRobot 数据集概览
在**任务规划（task planning）** 模块中，我们共收集了51403个轨迹片段，每个片段包含30帧图像。在数据生成阶段，我们针对RoboVQA（RoboVQA）[1]中的10类问题分别设计了5种不同的提问模板，并为每个样本随机选取2类模板生成问答对。该流程将51403个样本转化为1027990组问答对，同时由标注人员全程监控数据生成过程，以保障数据集的完整性。

在**物体功能属性（object affordance）** 模块中，我们共收录6522张图像，每张图像均配有与指令对应的功能属性标注区域。

在**末端执行器轨迹（end-effector trajectory）** 模块中，我们共收录6870张图像，每张图像均配有至少3组与指令对应的{x,y}坐标轨迹标注。

## 数据集来源


**ShareRobot 数据集**整合了源自Open X-Embodiment数据集（Open X-Embodiment dataset）[2]的23个原始数据集，涵盖12种机器人形态与107类原子任务。

### 任务规划原始数据集

| "原始数据集名称" | 样本数量 |
|:-------------------------------------------------------------:| --------------:|
| nyu_door_opening_surprising_effectiveness | 421 |
| bridge | 15738 |
| dlr_edan_shared_control_converted_externally_to_rlds | 63 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 92 |
| cmu_stretch | 10 |
| asu_table_top_converted_externally_to_rlds | 109 |
| dlr_sara_pour_converted_externally_to_rlds | 51 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 27 |
| robo_set | 18164 |
| dobbe | 5200 |
| berkeley_autolab_ur5 | 882 |
| qut_dexterous_manpulation | 192 |
| aloha_mobile | 264 |
| dlr_sara_grid_clamp_converted_externally_to_rlds | 40 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 569 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 39 |
| jaco_play | 956 |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 64 |
| conq_hose_manipulation | 56 |
| fmb | 7836 |
| plex_robosuite | 398 |
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 189 |
| viola | 44 |

### 物体功能属性原始数据集

| "原始数据集名称" | 样本数量 |
|:-------------------------------------------------------------:| -------------:|
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 24 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 23 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 10 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 112 |
| nyu_door_opening_surprising_effectiveness | 85 |
| jaco_play | 171 |
| bridge | 2610 |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 12 |
| asu_table_top_converted_externally_to_rlds | 24 |
| viola | 1 |
| berkeley_autolab_ur5 | 122 |
| aloha_mobile | 23 |
| conq_hose_manipulation | 1 |
| dobbe | 717 |
| fmb | 561 |
| plex_robosuite | 13 |
| qut_dexterous_manpulation | 16 |
| robo_set | 1979 |
| dlr_edan_shared_control_converted_externally_to_rlds | 18 |
| **总计** | 6522 |

### 末端执行器轨迹原始数据集

| "原始数据集名称" | 样本数量 |
|:-------------------------------------------------------------:| -------------:|
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 35 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 36 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 19 |
| dlr_sara_grid_clamp_converted_externally_to_rlds | 1 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 109 |
| nyu_door_opening_surprising_effectiveness | 74 |
| jaco_play | 175 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 7 |
| bridge | 2986 |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 12 |
| asu_table_top_converted_externally_to_rlds | 22 |
| berkeley_autolab_ur5 | 164 |
| dobbe | 759 |
| fmb | 48 |
| qut_dexterous_manpulation | 29 |
| robo_set | 2374 |
| dlr_sara_pour_converted_externally_to_rlds | 3 |
| dlr_edan_shared_control_converted_externally_to_rlds | 17 |
| **总计** | 6870 |

## 数据格式

### 任务规划格式


json
{
"id"{
"id": 0,
"task": "Future_Prediction_Task",
"selected_step": 3,
"conversations": [
{
"from": "human",
"value": "<image 0-25> After <move the grasped banana towards the mug>, what's the most probable next event?"
},
{
"from": "gpt",
"value": "<place the banana into the mug>"
}
],
"image": [
"/path/to/image_0-25"
]
}
}

### 物体功能属性格式

<div style="display: flex; gap: 10px;">
<img src="./images/2d94d985-d47e-4899-9760-c1cb8f19cd89.png" style="width: 300px;" />
<img src="./images/a7817c0b-04b1-4a7c-9535-f9ff7801a689.png" style="width: 300px;" />
</div>

json
{

"id": 2486,
"meta_data": {
"original_dataset": "bridge",
"original_width": 640,
"original_height": 480
},
"instruction": "place the red fork to the left of the left burner",
"affordance": {
"x": 352.87425387858815,
"y": 186.47871614766484,
"width": 19.296008229513156,
"height": 14.472006172134865
}

#### 可视化代码

python
import json
import os
import cv2
import numpy as np

img_dir = '/path/to/your/original/images/dir'
affordance_json = '/path/to/your/affordances/json'
output_img_dir = '/path/to/your/visualized/images/dir'

with open(affordance_json, 'r') as f:
data = json.load(f)
for item in data:
filepath = os.path.join(img_dir, item['id'])

image = cv2.imread(filepath)
color = (255, 0, 0)
thickness = 2

x_min,y_min = item['affordance']['x'], item['affordance']['y']
x_max,y_max = item['affordance']['x']+item['affordance']['width'], item['affordance']['y']+item['affordance']['height']

# 定义矩形的四个顶点坐标
pts = np.array([
[x_min, y_min], # 左上角
[x_max, y_min], # 右上角
[x_max, y_max], # 右下角
[x_min, y_max] # 左下角
], dtype=np.float32)

# 绘制矩形框
cv2.polylines(image, [pts.astype(int)], isClosed=True, color=color, thickness=thickness)

# 获取相对路径并拼接目标路径
relative_path = os.path.relpath(filepath, img_dir) # 获取相对于 img_dir 的相对路径
output_img_path = os.path.join(output_img_dir, relative_path) # 拼接目标路径

# 创建目标文件夹
output_directory = os.path.dirname(output_img_path)
if not os.path.exists(output_directory):
os.makedirs(output_directory)

# 打印调试信息
print(f"Input filepath: {filepath}")
print(f"Output image path: {output_img_path}")
print(f"Output directory: {output_directory}")

# 保存图像
cv2.imwrite(output_img_path, image)

### 末端执行器轨迹格式

<div style="display: flex; gap: 10px;">
<img src="./images/5b923b31-dbbf-470f-af09-5125f5b91ab0.png" style="width: 300px;" />
<img src="./images/1af4535a-acc3-4417-ae33-675f4301f560.png" style="width: 300px;" />
</div>

json
{
"id": 456,
"meta_data": {
"original_dataset": "bridge",
"original_width": 640,
"original_height": 480
},
"instruction": "reach for the carrot",
"points": [
[
265.45454545454544,
120.0
],
[
275.1515151515152,
162.42424242424244
],
[
280.0,
213.33333333333331
],
[
280.0,
259.3939393939394
]
]
},

#### 可视化代码

python
import json
import os
from PIL import Image, ImageDraw

trajectory_final = '/path/to/your/trajectory_json'
img_dir = '/path/to/your/original/images/dir'
output_img_dir = '/path/to/your/visualzed/images/dir'

with open(trajectory_final, 'r') as f:
data = json.load(f)
for item in data:
filepath = os.path.join(img_dir, item['id'])
points = item['points']

image = Image.open(filepath).convert("RGB") # 确保图像是 RGB 模式
draw = ImageDraw.Draw(image) # 创建绘图对象
# 定义颜色和线宽
color = (255, 0, 0) # 红色 (RGB 格式)
thickness = 2

scaled_points = [
(point[0], point[1])
for point in points
]
# 按照顺序连接相邻的点
for i in range(len(scaled_points) - 1):
draw.line([scaled_points[i], scaled_points[i + 1]], fill=color, width=thickness)

# 获取相对路径并拼接目标路径
relative_path = os.path.relpath(filepath, img_dir)
output_img_path = os.path.join(output_img_dir, relative_path)

# 创建目标文件夹
output_directory = os.path.dirname(output_img_path)
if not os.path.exists(output_directory):
os.makedirs(output_directory)

# 打印调试信息
print(f"Input filepath: {filepath}")
print(f"Output image path: {output_img_path}")
print(f"Output directory: {output_directory}")

# 保存图像
image.save(output_img_path)

## 评估

## 参考文献

[1] Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi等. RoboVQA：面向机器人的多模态长时序推理任务. 见：ICRA, 第645-652页, 2024.

[2] Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar等. Open X-Embodiment：机器人学习数据集与RT-X模型. arXiv预印本arXiv:2310.08864, 2023.

## 引用格式

@article{ji2025robobrain,
title={RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete},
author={Ji, Yuheng and Tan, Huajie and Shi, Jiayu and Hao, Xiaoshuai and Zhang, Yuan and Zhang, Hengyuan and Wang, Pengwei and Zhao, Mengdi and Mu, Yao and An, Pengju and others},
journal={arXiv preprint arXiv:2502.21257},
year={2025}
}