allenai/MolmoWeb-SyntheticSkills

---
dataset_info:
features:
- name: sample_id
dtype: string
- name: instruction
dtype: string
- name: trajectory
dtype: string
- name: images
sequence: image
- name: image_paths
sequence: string

splits:
- name: train
num_examples: 5535
- name: preview
num_examples: 10

configs:
- config_name: default
data_files:
- split: train
path: data/train-*.parquet
- split: preview
path: data/preview-00000.parquet
---

# MolmoWeb-SyntheticSkills

A dataset of synthetic web-navigation skills, where a skill is a trajectory for a very low level task (eg. find_and_open, fill_form). Each example pairs an instruction with a sequence of webpage screenshots and the corresponding agent actions (clicks, typing, scrolling, etc.).

## Dataset Usage

```python
from datasets import load_dataset

# load a single subset
ds = load_dataset("allenai/MolmoWeb-SyntheticSkills")
```

### Working with images and trajectories

Each row has an `images` field (list of raw image bytes) and a corresponding `image_paths` field (list of filenames). Use `image_paths` to match screenshots to trajectory steps:

```python
import json

row = ds[0]
traj = json.loads(row["trajectory"])

# build a lookup from filename -> image bytes
image_by_path = dict(zip(row["image_paths"], row["images"]))

for step_id in sorted(traj.keys(), key=int):
screenshot_name = traj[step_id].get("screenshot")
if not screenshot_name:
continue
img_bytes = image_by_path.get(screenshot_name)
# img_bytes is the raw PNG/JPEG data for this step
```

## Dataset Structure

### Features

| Field | Type | Description |
|---|---|---|
| `sample_id` | `string` | Unique hash identifying the trajectory |
| `instruction` | `string` | JSON-encoded task instruction (contains a `low_level` key or similar) |
| `trajectory` | `string` | JSON-encoded trajectory: a dict keyed by step index, each entry containing the agent's parsed action and screenshot filename |
| `images` | `list[bytes]` | List of screenshot structs; `bytes` is the raw image data, `path` is the filename used to match against trajectory steps |
| `image_paths` | `list[path]` | List of paths to screenshots used to match against trajectory steps |

### Trajectory step structure

Each step in `trajectory` (keyed by step index) contains:

| Field | Type | Description |
|---|---|---|
| `screenshot` | `string` | Filename matching an entry in the `images` list |
| `action` | `dict` | The agent action: `action_str` (parseable action string), `action_description` (natural language), and `action_output` (structured dict with `thought`, `action_name`, and action parameters) |
| `other_obs` | `dict` | Browser state: current `url`, `page_index`, `open_pages_titles`, `open_pages_urls` |
| `action_timestamp` | `float` | Unix timestamp of the action |

## License

This dataset is licensed under ODC-BY 1.0. It is intended for research and educational use in accordance with [Ai2's Responsible Use Guidelines](https://allenai.org/responsible-use). Instruction data was generated using Gemini Flash models (subject to [Google's Terms of Service](https://policies.google.com/terms)) and GPT models (subject to [OpenAI's Terms of Use](https://openai.com/policies/row-terms-of-use/)).

dataset_info: 数据集信息：
features:
- name: sample_id
dtype: 字符串
- name: instruction
dtype: 字符串
- name: trajectory
dtype: 字符串
- name: images
dtype: 图像序列
- name: image_paths
dtype: 字符串序列

splits:
- name: train
num_examples: 5535
- name: preview
num_examples: 10

configs:
- config_name: default
data_files:
- split: train
path: data/train-*.parquet
- split: preview
path: data/preview-00000.parquet

---

# MolmoWeb-SyntheticSkills

这是一款合成网页导航技能数据集，其中“技能”指面向极细粒度任务的AI智能体轨迹 (AI Agent trajectory)，例如find_and_open、fill_form。每个样本将任务指令与一系列网页截图以及对应的AI智能体操作（点击、输入、滚动等）进行配对。

## 数据集使用方法

python
from datasets import load_dataset

# 加载单个子集
ds = load_dataset("allenai/MolmoWeb-SyntheticSkills")

### 处理图像与智能体轨迹

每个样本均包含`images`字段（原始图像字节列表）与对应的`image_paths`字段（文件名列表）。可通过`image_paths`将截图与轨迹步骤进行匹配：

python
import json

row = ds[0]
traj = json.loads(row["trajectory"])

# 构建文件名到图像字节的映射表
image_by_path = dict(zip(row["image_paths"], row["images"]))

for step_id in sorted(traj.keys(), key=int):
screenshot_name = traj[step_id].get("screenshot")
if not screenshot_name:
continue
img_bytes = image_by_path.get(screenshot_name)
# img_bytes 即为当前步骤的原始PNG/JPEG图像数据

## 数据集结构

### 特征字段

| 字段 | 类型 | 说明 |
|---|---|---|
| `sample_id` | `string` | 用于标识轨迹的唯一哈希值 |
| `instruction` | `string` | JSON编码的任务指令，包含`low_level`等键 |
| `trajectory` | `string` | JSON编码的轨迹：以步骤索引为键的字典，每个条目包含智能体解析后的操作与截图文件名 |
| `images` | `list[bytes]` | 截图结构体列表；`bytes`为原始图像数据，`path`为用于匹配轨迹步骤的文件名 |
| `image_paths` | `list[path]` | 用于匹配轨迹步骤的截图路径列表 |

### 轨迹步骤结构

`trajectory`中的每个步骤（以步骤索引为键）包含以下字段：

| 字段 | 类型 | 说明 |
|---|---|---|
| `screenshot` | `string` | 与`images`列表中某条目匹配的文件名 |
| `action` | `dict` | 智能体操作：包含`action_str`（可解析的操作字符串）、`action_description`（自然语言描述）以及`action_output`（结构化字典，包含`thought`、`action_name`与操作参数） |
| `other_obs` | `dict` | 浏览器状态：当前`url`、`page_index`、`open_pages_titles`（已打开页面标题）、`open_pages_urls`（已打开页面URL） |
| `action_timestamp` | `float` | 操作对应的Unix时间戳 |

## 许可证

本数据集采用ODC-BY 1.0许可证，仅可用于符合[艾伦人工智能研究所（AI2）负责任使用指南](https://allenai.org/responsible-use)的研究与教育用途。指令数据通过Gemini Flash模型（受[Google服务条款](https://policies.google.com/terms)约束）与GPT模型（受[OpenAI使用条款](https://openai.com/policies/row-terms-of-use/)约束）生成。