Sheelu1246/MolmoWeb-HumanSkills

---
dataset_info:
features:
- name: sample_id
dtype: string
- name: instruction
dtype: string
- name: trajectory
dtype: string
- name: images
sequence: image
- name: image_paths
sequence: string

splits:
- name: train
num_examples: 115637
- name: preview
num_examples: 10

configs:
- config_name: default
data_files:
- split: train
path: data/train-*.parquet
- split: preview
path: data/preview-00000.parquet
---

# MolmoWeb-HumanSkills

A dataset of human collected web-navigation skills, where a skill is a trajectory for a very low level task (eg. find_and_open, fill_form). Each example pairs an instruction with a sequence of webpage screenshots and the corresponding agent actions (clicks, typing, scrolling, etc.).

## Dataset Usage

```python
from datasets import load_dataset

# load a single subset
ds = load_dataset("allenai/MolmoWeb-HumanSkills")
```

### Working with images and trajectories

Each row has an `images` field (list of raw image bytes) and a corresponding `image_paths` field (list of filenames). Use `image_paths` to match screenshots to trajectory steps:

```python
import json

row = ds[0]
traj = json.loads(row["trajectory"])

# build a lookup from filename -> image bytes
image_by_path = dict(zip(row["image_paths"], row["images"]))

for step_id in sorted(traj.keys(), key=int):
screenshot_name = traj[step_id].get("screenshot")
if not screenshot_name:
continue
img_bytes = image_by_path.get(screenshot_name)
# img_bytes is the raw PNG/JPEG data for this step
```

## Dataset Structure

### Features

| Field | Type | Description |
|---|---|---|
| `sample_id` | `string` | Unique hash identifying the trajectory |
| `instruction` | `string` | JSON-encoded task instruction (contains a `low_level` key or similar) |
| `trajectory` | `string` | JSON-encoded trajectory: a dict keyed by step index, each entry containing the agent's parsed action and screenshot filename |
| `images` | `list[bytes]` | List of screenshot structs; `bytes` is the raw image data, `path` is the filename used to match against trajectory steps |
| `image_paths` | `list[path]` | List of paths to screenshots used to match against trajectory steps |

### Trajectory step structure

Each step in `trajectory` (keyed by step index) contains:

| Field | Type | Description |
|---|---|---|
| `screenshot` | `string` | Filename matching an entry in the `images` list |
| `action` | `dict` | The agent action: `action_str` (parseable action string), `action_description` (natural language), and `action_output` (structured dict with `thought`, `action_name`, and action parameters) |
| `other_obs` | `dict` | Browser state: current `url`, `page_index`, `open_pages_titles`, `open_pages_urls` |
| `action_timestamp` | `float` | Unix timestamp of the action |

## License

This dataset is licensed under ODC-BY 1.0. It is intended for research and educational use in accordance with [Ai2's Responsible Use Guidelines](https://allenai.org/responsible-use). Instruction data was generated using GPT models, which are subject to [OpenAI's Terms of Use](https://openai.com/policies/row-terms-of-use/).

---
数据集信息：
特征：
- 名称：sample_id（样本ID），数据类型：字符串
- 名称：instruction（指令），数据类型：字符串
- 名称：trajectory（轨迹），数据类型：字符串
- 名称：images（图像），序列类型：图像
- 名称：image_paths（图像路径），序列类型：字符串

划分：
- 名称：train（训练集），样本数量：115637
- 名称：preview（预览集），样本数量：10

配置：
- 配置名称：default（默认配置），数据文件：
- 划分：train，路径：data/train-*.parquet
- 划分：preview，路径：data/preview-00000.parquet
---

# MolmoWeb-HumanSkills：人类采集的网页导航技能数据集
该数据集收录了人类收集的网页导航技能，其中技能指针对极细粒度任务的操作轨迹（例如find_and_open、fill_form）。每个样本将任务指令与网页截图序列及对应的AI智能体（AI Agent）操作（点击、输入、滚动等）进行配对。

## 数据集使用方法
python
from datasets import load_dataset

# 加载单个子集
ds = load_dataset("allenai/MolmoWeb-HumanSkills")

### 图像与轨迹处理
每条样本均包含`images`字段（原始图像字节列表）与对应的`image_paths`字段（文件名列表）。可通过`image_paths`将截图与轨迹步骤进行匹配：
python
import json

row = ds[0]
traj = json.loads(row["trajectory"])

# 构建文件名到图像字节的映射表
image_by_path = dict(zip(row["image_paths"], row["images"]))

for step_id in sorted(traj.keys(), key=int):
screenshot_name = traj[step_id].get("screenshot")
if not screenshot_name:
continue
img_bytes = image_by_path.get(screenshot_name)
# img_bytes 即为当前步骤的原始PNG/JPEG图像数据

## 数据集结构
### 特征字段
| 字段 | 类型 | 描述 |
|---|---|---|
| `sample_id` | `string` | 用于唯一标识轨迹的哈希值 |
| `instruction` | `string` | JSON编码的任务指令，包含`low_level`等键 |
| `trajectory` | `string` | JSON编码的轨迹：以步骤索引为键的字典，每个条目包含智能体解析后的操作与截图文件名 |
| `images` | `list[bytes]` | 截图结构体列表；`bytes`为原始图像数据，`path`为用于匹配轨迹步骤的文件名 |
| `image_paths` | `list[path]` | 用于匹配轨迹步骤的截图路径列表 |

### 轨迹步骤结构
轨迹中的每个步骤（以步骤索引为键）包含以下字段：
| 字段 | 类型 | 描述 |
|---|---|---|
| `screenshot` | `string` | 与`images`列表中条目匹配的文件名 |
| `action` | `dict` | 智能体操作字典：包含`action_str`（可解析的操作字符串）、`action_description`（自然语言描述）与`action_output`（结构化字典，包含`thought`、`action_name`及操作参数） |
| `other_obs` | `dict` | 浏览器状态信息：包含当前`url`、`page_index`、`open_pages_titles`（已打开页面标题）与`open_pages_urls`（已打开页面URL） |
| `action_timestamp` | `float` | 操作对应的Unix时间戳 |

## 许可证
本数据集采用ODC-BY 1.0许可证进行授权，仅可用于研究与教育用途，并需遵循[AI2负责任使用指南](https://allenai.org/responsible-use)。指令数据通过GPT模型生成，需受[OpenAI使用条款](https://openai.com/policies/row-terms-of-use/)约束。