GUI-Robust
收藏资源简介：
GUI-Robust是一个用于评估图形用户界面(GUI)代理在现实世界异常情况下的鲁棒性的综合数据集。该数据集包含5318个带注释的任务，来自392个不同的来源，包括网站和Windows桌面应用程序。特别地，它包括200个异常任务，涵盖7种日常GUI使用中常见的异常类型，如动作失败、登录页面、验证码页面、广告弹出、cookie弹出、页面加载和网络断开。GUI-Robust提供了广泛的任务和动作类型，包括点击、输入文本、从页面检索信息、打开新的网页或应用程序以及向人类报告异常。此外，它还涵盖了跨场景任务，跨越多个应用程序或网站，反映了更真实和复杂的工作流程，并覆盖了中文和英文软件环境。
GUI-Robust is a comprehensive dataset designed to evaluate the robustness of graphical user interface (GUI) agents under real-world anomalous scenarios. This dataset comprises 5,318 annotated tasks sourced from 392 distinct origins, including websites and Windows desktop applications. Specifically, it contains 200 anomalous tasks covering 7 common anomaly types frequently seen in daily GUI operations, such as action failures, login pages, CAPTCHA pages, advertisement pop-ups, cookie pop-ups, page loading anomalies, and network disconnections. GUI-Robust offers a diverse array of task and action types, encompassing clicking, text input, information retrieval from web pages, launching new webpages or applications, and reporting anomalies to human users. Furthermore, it includes cross-scenario tasks that span multiple applications or websites, mirroring more realistic and complex workflows, and supports both Chinese and English software environments.
GUI-Robust数据集概述
数据集简介
- 名称: GUI-Robust
- 用途: 用于测试GUI代理在现实世界异常情况下的鲁棒性
- 完整数据集地址: https://huggingface.co/datasets/kuangtie/GUI-Robust
评估脚本
-
运行命令: bash python evaluation.py --model_name <YourModel> --eval_type step|task --task_type normal|abnormal --data_path <path_to_data_folder>
-
评估模式:
step: 评估每步的基础准确性（动作准确性和坐标准确性）task: 评估完整任务执行（动作准确性、坐标准确性和任务成功率）
模型集成
接口规范
-
单步预测方法: python def pred_step_loc(step_description: str, screenshot_base64: str) -> dict
-
全任务预测方法: python def pred_task_full(task_description: str, screenshot_list_base64: List[str]) -> List[dict]
预测输出格式
- 元素坐标 (x, y)
- 元素类型 (icon, text, box, none)
- 动作类型及内容 (click, input, get_info, open, wait, human)
引用
bibtex @inproceedings{ yang2025guirobust, title={GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies}, author={Jingqi Yang and Zhilong Song and Jiawei Chen and Mingli Song and Sheng Zhou and Linjun Sun and Xiaogang Ouyang and Chun Chen and Can Wang}, booktitle={NeurIPS Datasets and Benchmarks Track}, year={2025}, url={https://openreview.net/forum?id=22gw3kITCd}, }
