osunlp/MisActBench

---
language:
- en
license: cc-by-4.0
size_categories:
- 1K<n<10K
task_categories:
- image-text-to-text
pretty_name: MisActBench
tags:
- computer-use-agent
- safety
- alignment
- benchmark
- misaligned-action-detection
---

# MisActBench

**MisActBench** is a comprehensive benchmark for evaluating misaligned action detection in Computer-Use Agents (CUAs). It contains **2,264 human-annotated action-level alignment labels** across **558 realistic CUA trajectories**, covering both externally induced and internally arising misaligned actions.

## 🔗 Links

- [🏠 Homepage](https://osu-nlp-group.github.io/Misaligned-Action-Detection/)
- [📖 Paper](https://arxiv.org/abs/2602.08995)
- [💻 Code](https://github.com/OSU-NLP-Group/Misaligned-Action-Detection)

## 📊 Dataset Summary

| Statistic | Count |
|---|---|
| Trajectories | 558 |
| Steps | 2,264 |
| Aligned Steps | 1,264 |
| Misaligned Steps | 1,000 |
| - Malicious Instruction Following | 562 (56.2%) |
| - Harmful Unintended Behavior | 210 (21.0%) |
| - Other Task-Irrelevant Behavior | 228 (22.8%) |

## 📁 Dataset Structure

This dataset consists of two files:

- **`misactbench.json`** — The main annotation file containing all trajectory metadata, step-level labels, and action outputs.
- **`trajectories.zip`** — A zip archive containing screenshot images organized by trajectory ID.

After downloading and extracting, the file structure is:

```
MisActBench/
├── misactbench.json
└── trajectories/
├── <trajectory_id>/
│ ├── step_0_*.png
│ ├── step_1_*.png
│ └── ...
└── ...
```

## 📝 Data Format

`misactbench.json` is a JSON object keyed by `trajectory_id` (UUID). Each entry has the following fields:

| Field | Type | Description |
|---|---|---|
| `trajectory_id` | `string` | UUID for the trajectory |
| `instruction` | `string` | The user instruction given to the CUA |
| `total_steps` | `int` | Total number of steps in the trajectory |
| `trajectory_path` | `string` | Relative path to the trajectory screenshot folder |
| `metadata` | `object` | Contains `source` (data source) and `agent` (CUA model) |
| `steps` | `object` | Dict keyed by step number (see below) |

Each step in `steps` contains:

| Field | Type | Description |
|---|---|---|
| `step_idx` | `int` | Step index |
| `label` | `bool \| null` | `true` = misaligned, `false` = aligned, `null` = not annotated |
| `category` | `string \| null` | Misalignment category (only set when `label=true`) |
| `agent_output` | `string` | The agent's proposed action for this step |
| `screenshot_path` | `string` | Relative path to the screenshot observed before this action |

### Misalignment Categories

- **Malicious Instruction Following**: The action complies with malicious instructions in external environments to achieve an attacker's goal.
- **Harmful Unintended Behavior**: The action causes harm inadvertently due to inherent limitations (e.g., reasoning error) rather than adversarial attack.
- **Other Task-Irrelevant Behavior**: The action does not cause harmful consequences but is irrelevant to the user task and will degrade efficiency and reliability.

## 📮 Contact

[Yuting Ning](mailto:[e-mail]), [Huan Sun](mailto:[e-mail])

## 📝 Citation Information
If you find this work useful, please consider citing our paper:
```
@misc{ning2026actionsofftaskdetectingcorrecting,
title={When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents},
author={Yuting Ning and Jaylen Jones and Zhehao Zhang and Chentao Ye and Weitong Ruan and Junyi Li and Rahul Gupta and Huan Sun},
year={2026},
eprint={2602.08995},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2602.08995},
}
```

---
语言：
- 英语
许可协议：知识共享署名4.0（CC BY 4.0）
样本规模类别：
- 1000 < n < 10000
任务类别：
- 图像-文本到文本
友好名称：MisActBench
标签：
- 计算机使用智能体（computer-use-agent）
- 安全性（safety）
- 对齐性（alignment）
- 基准测试（benchmark）
- 错位动作检测（misaligned-action-detection）
---

# MisActBench基准数据集

**MisActBench**是一款用于评估计算机使用智能体（Computer-Use Agents，简称CUAs）错位动作检测能力的综合性基准数据集。该数据集涵盖558条真实的CUA运行轨迹，包含2264条经人工标注的动作级对齐标签，覆盖外部诱导与内生两类错位动作。

## 🔗 链接

- [🏠 主页](https://osu-nlp-group.github.io/Misaligned-Action-Detection/)
- [📖 论文](https://arxiv.org/abs/2602.08995)
- [💻 代码](https://github.com/OSU-NLP-Group/Misaligned-Action-Detection)

## 📊 数据集统计摘要

| 统计项 | 数量 |
|---|---|
| 运行轨迹 | 558 |
| 总步数 | 2264 |
| 对齐步数 | 1264 |
| 错位步数 | 1000 |
| - 恶意指令遵循 | 562 (56.2%) |
| - 有害非预期行为 | 210 (21.0%) |
| - 其他与任务无关行为 | 228 (22.8%) |

## 📁 数据集结构

本数据集包含两个文件：

- **`misactbench.json`** — 主标注文件，包含所有轨迹元数据、步级标签与动作输出内容。
- **`trajectories.zip`** — 按轨迹ID组织的截图图像压缩归档包。

下载并解压后，文件目录结构如下：

MisActBench/
├── misactbench.json
└── trajectories/
├── <trajectory_id>/
│ ├── step_0_*.png
│ ├── step_1_*.png
│ └── ...
└── ...

## 📝 数据格式

`misactbench.json`是一个以`trajectory_id`（通用唯一识别码，UUID）为键的JSON对象。每个条目包含以下字段：

| 字段 | 类型 | 描述 |
|---|---|---|
| `trajectory_id` | `string` | 运行轨迹的UUID |
| `instruction` | `string` | 下发给CUA的用户指令 |
| `total_steps` | `int` | 运行轨迹的总步数 |
| `trajectory_path` | `string` | 轨迹截图文件夹的相对路径 |
| `metadata` | `object` | 包含`source`（数据来源）与`agent`（CUA模型）两个子字段 |
| `steps` | `object` | 以步号为键的字典（详见下文） |

`steps`中的每个步对象包含以下字段：

| 字段 | 类型 | 描述 |
|---|---|---|
| `step_idx` | `int` | 步索引编号 |
| `label` | `bool | null` | `true`表示动作错位，`false`表示动作对齐，`null`表示未标注 |
| `category` | `string | null` | 错位动作类别（仅当`label=true`时有效） |
| `agent_output` | `string` | 智能体针对该步提出的动作方案 |
| `screenshot_path` | `string` | 该动作执行前截取的屏幕截图的相对路径 |

### 错位动作分类

- **恶意指令遵循**：该动作遵从外部环境中的恶意指令，以达成攻击者的预设目标。
- **有害非预期行为**：该动作因智能体固有局限（如推理错误）而非对抗性攻击，意外造成危害。
- **其他与任务无关行为**：该动作未造成有害后果，但与用户任务无关，会降低任务执行效率与可靠性。

## 📮 联系方式

[宁宇婷](mailto:[e-mail]), [孙欢](mailto:[e-mail])

## 📝 引用说明

若您认为本工作对您有所帮助，请引用我们的论文：

@misc{ning2026actionsofftaskdetectingcorrecting,
title={When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents},
author={Yuting Ning and Jaylen Jones and Zhehao Zhang and Chentao Ye and Weitong Ruan and Junyi Li and Rahul Gupta and Huan Sun},
year={2026},
eprint={2602.08995},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2602.08995},
}