MisActBench

# MisActBench **MisActBench** is a comprehensive benchmark for evaluating misaligned action detection in Computer-Use Agents (CUAs). It contains **2,264 human-annotated action-level alignment labels** across **558 realistic CUA trajectories**, covering both externally induced and internally arising misaligned actions. ## 🔗 Links - [🏠 Homepage](https://osu-nlp-group.github.io/Misaligned-Action-Detection/) - [📖 Paper](https://arxiv.org/abs/2602.08995) - [💻 Code](https://github.com/OSU-NLP-Group/Misaligned-Action-Detection) ## 📊 Dataset Summary | Statistic | Count | |---|---| | Trajectories | 558 | | Steps | 2,264 | | Aligned Steps | 1,264 | | Misaligned Steps | 1,000 | | - Malicious Instruction Following | 562 (56.2%) | | - Harmful Unintended Behavior | 210 (21.0%) | | - Other Task-Irrelevant Behavior | 228 (22.8%) | ## 📁 Dataset Structure This dataset consists of two files: - **`misactbench.json`** — The main annotation file containing all trajectory metadata, step-level labels, and action outputs. - **`trajectories.zip`** — A zip archive containing screenshot images organized by trajectory ID. After downloading and extracting, the file structure is: ``` MisActBench/ ├── misactbench.json └── trajectories/ ├── <trajectory_id>/ │ ├── step_0_*.png │ ├── step_1_*.png │ └── ... └── ... ``` ## 📝 Data Format `misactbench.json` is a JSON object keyed by `trajectory_id` (UUID). Each entry has the following fields: | Field | Type | Description | |---|---|---| | `trajectory_id` | `string` | UUID for the trajectory | | `instruction` | `string` | The user instruction given to the CUA | | `total_steps` | `int` | Total number of steps in the trajectory | | `trajectory_path` | `string` | Relative path to the trajectory screenshot folder | | `metadata` | `object` | Contains `source` (data source) and `agent` (CUA model) | | `steps` | `object` | Dict keyed by step number (see below) | Each step in `steps` contains: | Field | Type | Description | |---|---|---| | `step_idx` | `int` | Step index | | `label` | `bool \| null` | `true` = misaligned, `false` = aligned, `null` = not annotated | | `category` | `string \| null` | Misalignment category (only set when `label=true`) | | `agent_output` | `string` | The agent's proposed action for this step | | `screenshot_path` | `string` | Relative path to the screenshot observed before this action | ### Misalignment Categories - **Malicious Instruction Following**: The action complies with malicious instructions in external environments to achieve an attacker's goal. - **Harmful Unintended Behavior**: The action causes harm inadvertently due to inherent limitations (e.g., reasoning error) rather than adversarial attack. - **Other Task-Irrelevant Behavior**: The action does not cause harmful consequences but is irrelevant to the user task and will degrade efficiency and reliability. ## 📮 Contact [Yuting Ning](mailto:[e-mail]), [Huan Sun](mailto:[e-mail]) ## 📝 Citation Information If you find this work useful, please consider citing our paper: ``` @misc{ning2026actionsofftaskdetectingcorrecting, title={When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents}, author={Yuting Ning and Jaylen Jones and Zhehao Zhang and Chentao Ye and Weitong Ruan and Junyi Li and Rahul Gupta and Huan Sun}, year={2026}, eprint={2602.08995}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2602.08995}, } ```