VeriWeb
收藏资源简介：
<h1 align="center">VeriWeb: Verifiable Long-Chain Web Benchmark for Agentic Information-Seeking</h1> <div align="center"> <a href='https://huggingface.co/papers/2508.04026'><img src='https://img.shields.io/badge/Paper-Arxiv-red.svg?style=for-the-badge&logo=arxiv&logoColor=white'></a> <a href='https://huggingface.co/datasets/2077AIDataFoundation/VeriGUI'><img src='https://img.shields.io/badge/Dataset-Hugging_Face-yellow.svg?style=for-the-badge&logo=huggingface&logoColor=%23FFD21E'></a> <a href='LICENSE'><img src='https://img.shields.io/badge/License-Apache_2.0-blue.svg?style=for-the-badge'></a> </div> > [!NOTE] > This project was originally named VeriGUI. As our initial data collection focused on web-based tasks that primarily involve information-seeking rather than GUI interaction, we now define this part as the standalone VeriWeb benchmark, while desktop and other GUI-oriented scenarios will be released as a separate benchmark (in progress). We apologize for any resulting confusion. # Overview VeriWeb is a novel verifiable long-chain web benchmark designed to facilitate the evaluation and development of web agents within realistic web environments. Unlike existing efforts that mainly focus on single-fact retrieval and rely on outcome-only verification, VeriWeb emphasizes **long-chain complexity** and **subtask-level verifiability** to better reflect realistic knowledge-intensive scenarios. ## Updates - `[Jan 21, 2026]` Fixed and updated data: 658 659 662 665 683 687 698 701 703 743 744 750 V3_80 V2_12 V5_61 V6_20 - `[Oct 23, 2025]` 🔥 We have released the updated 302 web-based task trajectories! - `[Jul 21, 2025]` 🔥 We have released the first batch of 130 web-based task trajectories! ## Key Features ### 🔗 Long-Chain Complexity - 302 realistic information-seeking tasks across 5 real-world domains - Long-chain web trajectories decomposed into multiple interdependent subtasks - Tasks combine breadth-oriented search and depth-oriented search - Agents must retrieve, reason, and synthesize evidence from diverse web pages ### ✅ Subtask-Level Verifiability - Fine-grained evaluation at each intermediate subtask, not only the final outcome - Fixed, verifiable target outputs for every subtask while supporting diverse exploration strategies - Each subtask can serve as an independent starting point, enabling evaluation at different stages of a task - Rich supervision signals for diagnosing failure modes ### 🧑🎨 Human-Expert Annotation - All tasks and trajectories carefully created and annotated by human experts - High-quality task instructions, subtask decompositions, and answer annotations - Each task includes a complete human demonstration with detailed observation and action logs # Visualize Tool ## Usage 1. Open [VeriGUI.2077ai.org](https://verigui.2077ai.org) 2. Select the corresponding task data folder 3. View the visualization results ## Features - Interactive event timeline visualization - Support for various event types (MOUSE_DRAG, MOUSE_UP, TAB_CHANGE, etc.) - Video playback synchronization - Jump to specific actions functionality # Dataset Structure ``` VeriGUI/ ├── task_001/ │ ├── data.json # Complete task annotation │ └── video.mp4 # Video recording of task execution └── task_002/ ├── data.json └── video.mp4 ``` ## Task Structure ```json 📋 Complete Task ├── instruct (String): Complete task instruction ├── result (String): Final answer for the complete task └── actions (Array): List of subtask │ └── 📝 Subtask ├── instruct (String): Subtask-level instruction ├── result (String): Subtask-level answer for the subtask └── innerActions (Array): Low-level GUI actions within this step │ └── 🖱️ Action Object ├── type (String): Type of GUI action ├── url (String): Current webpage URL ├── rawHtml (String): Raw HTML content (optional) ├── time (Integer): Timestamp in milliseconds └── info (Object): Detailed action information ├── clientX/Y (Integer): Mouse coordinates relative to viewport ├── pageX/Y (Integer): Mouse coordinates relative to page ├── layerX/Y (Integer): Mouse coordinates relative to layer ├── screenX/Y (Integer): Mouse coordinates relative to screen ├── offsetX/Y (Integer): Mouse coordinates relative to target element ├── altKey/shiftKey/metaKey (Boolean): Modifier key states └── target (Object): Target DOM element information ├── innerText (String): Text content of target element ├── className (String): CSS class name └── [other DOM properties] ``` # Citation If you use VeriWeb in your research, please cite: ``` @article{verigui2025, title={VeriGUI: Verifiable Long-Chain GUI Dataset}, author={Shunyu Liu, Minghao Liu, Huichi Zhou, Zhenyu Cui, Yang Zhou, Yuhao Zhou, Wendong Fan, Ge Zhang, Jiajun Shi, Weihao Xuan, Jiaxing Huang, Shuang Luo, Fang Wu, Heli Qi, Qingcheng Zeng, Ziqi Ren, Jialiang Gao, Jindi Lv, Junjie Wang, Aosong Feng, Heng Zhou, Wangchunshu Zhou, Zhenfei Yin, Wenlong Zhang, Guohao Li, Wenhao Yu, Irene Li, Lei Ma, Lei Bai, Qunshu Lin, Mingli Song, Dacheng Tao}, journal={arXiv preprint arXiv:2508.04026}, year={2025} } ``` # License This dataset is released under the Apache-2.0
<div align="center"> # VeriGUI：可验证长序列图形用户界面数据集 </div> <div align="center"> <a href='https://huggingface.co/papers/2508.04026'><img src='https://img.shields.io/badge/Paper-Arxiv-red.svg?style=for-the-badge&logo=arxiv&logoColor=white'></a> <a href='https://huggingface.co/datasets/2077AIDataFoundation/VeriGUI'><img src='https://img.shields.io/badge/Dataset-Hugging_Face-yellow.svg?style=for-the-badge&logo=huggingface&logoColor=%23FFD21E'></a> <a href='LICENSE'><img src='https://img.shields.io/badge/License-Apache_2.0-blue.svg?style=for-the-badge'></a> </div> # 概览 VeriGUI是一个大规模人工标注数据集，旨在助力能够在真实计算机环境中完成复杂长周期任务的自主图形用户界面（Graphical User Interface, GUI）智能体（AI Agent）的开发与评估。与现有聚焦短期交互的GUI数据集不同，VeriGUI着重强调**长序列复杂性**与**子任务级可验证性**，以更贴合真实人机交互场景。 ## 更新日志 - `[2025年10月23日]` 🔥 我们已发布更新后的302条网页任务轨迹！ - `[2025年7月21日]` 🔥 我们已发布首批130条网页任务轨迹！ ## 核心特性 ### 🔗 长序列复杂性 - 任务包含**2至15个相互关联的子任务**，涉及数百次GUI操作 - 覆盖多应用与多网页的复杂工作流 - 贴合真实场景的任务依赖关系，需要智能体进行自适应推理与规划 - 任务完全复刻真实计算机使用模式 ### ✅ 子任务级可验证性 - 在每个中间子任务层面开展**细粒度评估**，而非仅关注最终结果 - 为每个子任务设置可验证的目标，同时支持多样化探索策略 - 支持子任务内的开放式交互——智能体可通过不同路径达成同一目标 - 提供详细监督信号，便于更好地进行错误诊断与智能体性能优化 ### 🌐 多环境覆盖 - **网页环境**：涵盖各类网站、在线服务与网页应用 - **桌面环境**：包含办公软件、操作系统与专业工具（待完成） - 支持跨平台任务切换与交互 ### 🧑🎨 专家人工标注 - 所有任务轨迹均由人类专家精心创建与标注 - 提供高质量任务指令与子任务级标注 - 验证了任务可行性与真实工作流模式 # 基准测试排行榜 ## 深度研究智能体 | 方法 | 科研 | | 金融 | | 科技 | | 艺术 | | 社交 | | 平均 | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | | OpenAI-o3 | 12.5 | **31.9** | 0.0 | 18.7 | **10.0** | **26.3** | **16.1** | **43.9** | 3.3 | **21.7** | **8.5** | **28.8** | | OpenAI-o4-mini | 0.0 | 8.1 | 0.0 | 17.0 | 6.7 | 20.7 | 12.9 | 30.6 | 3.3 | 19.0 | 5.4 | 20.5 | | Gemini-2.5-Flash | 6.2 | 19.4 | 0.0 | 14.3 | 3.3 | 16.7 | **16.1** | 41.0 | **6.7** | 17.7 | 6.9 | 22.6 | | Gemini-2.5-Pro | **18.8** | **31.9** | 0.0 | **22.2** | **10.0** | 23.7 | **16.1** | 41.6 | 0.0 | 21.0 | **8.5** | 28.1 | ## 搜索引擎智能体 | 方法 | 科研 | | 金融 | | 科技 | | 艺术 | | 社交 | | 平均 | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | | GPT-4o | 0.0 | 3.1 | 0.0 | 3.0 | 3.3 | 10.3 | 0.0 | 3.9 | 0.0 | 4.3 | 0.8 | 5.2 | | GPT-4.1 | 0.0 | **13.1** | 0.0 | **14.8** | 3.3 | 14.3 | 9.7 | 23.5 | 0.0 | 8.0 | 3.1 | 15.0 | | OpenAI-o3 | 0.0 | 5.0 | 0.0 | 13.5 | 10.0 | 19.0 | **12.9** | **35.2** | 0.0 | **11.0** | **5.4** | **18.3** | | Gemini-2.5-Flash | 0.0 | 5.0 | 0.0 | 7.4 | 0.0 | 8.3 | 6.5 | 28.1 | 0.0 | 6.7 | 1.5 | 12.1 | | Gemini-2.5-Pro | 0.0 | 4.4 | 0.0 | 8.7 | 3.3 | 12.0 | **12.9** | 28.1 | 0.0 | 7.7 | 3.8 | 13.3 | | Claude-3.7-Sonnet | 0.0 | 8.1 | 0.0 | 10.9 | **13.3** | **23.7** | 9.7 | 30.0 | 0.0 | 8.0 | **5.4** | 17.4 | | Claude-4.0-Sonnet | 0.0 | 11.9 | 0.0 | 11.3 | 6.7 | 13.7 | **12.9** | 21.9 | 0.0 | **11.0** | 4.6 | 14.4 | | Deepseek-Chat | 0.0 | 4.4 | 0.0 | 2.2 | 3.3 | 10.7 | **12.9** | 24.8 | 0.0 | 4.7 | 3.8 | 10.4 | ## 浏览器交互智能体 | 方法 | 科研 | | 金融 | | 科技 | | 艺术 | | 社交 | | 平均 | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | | GPT-4o | 0.0 | 1.9 | 0.0 | 1.7 | 3.3 | 8.3 | 3.2 | 13.5 | 0.0 | 5.7 | 1.5 | 7.0 | | GPT-4.1 | 0.0 | 3.8 | 0.0 | 7.0 | 3.3 | 9.0 | 16.1 | 29.7 | 0.0 | 9.7 | 4.6 | 13.1 | | OpenAI-o3 | **6.2** | **20.6** | 0.0 | **11.3** | 0.0 | **18.7** | 16.1 | 33.5 | 0.0 | **12.3** | 4.6 | **19.7** | | Gemini-2.5-Flash | 0.0 | 1.9 | 0.0 | 6.1 | 0.0 | 2.0 | 0.0 | 19.7 | 0.0 | 7.3 | 0.0 | 8.2 | | Gemini-2.5-Pro | **6.2** | 10.6 | 0.0 | 6.1 | **6.7** | 9.7 | 12.9 | 36.1 | 0.0 | 10.0 | 5.4 | 15.5 | | Claude-3.7-Sonnet | 0.0 | 7.5 | 0.0 | 9.6 | 0.0 | 15.3 | 16.1 | 36.8 | 0.0 | 10.3 | 3.8 | 17.3 | | Claude-4.0-Sonnet | **6.2** | 13.8 | 0.0 | 6.5 | 0.0 | 11.3 | **19.4** | **45.8** | **3.3** | 9.3 | **6.2** | 18.5 | | Qwen-VL-Max | 0.0 | 2.5 | 0.0 | 0.9 | 0.0 | 3.0 | 6.5 | 11.6 | 0.0 | 4.3 | 1.5 | 4.9 | ## 多智能体系统 | 方法 | 科研 | | 金融 | | 科技 | | 艺术 | | 社交 | | 平均 | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | **成功率（%）** | **完成率（%）** | | OWL with OpenAI-o3 | 6.2 | 18.8 | 0.0 | 6.5 | 3.3 | 11.3 | 16.1 | 32.3 | 6.7 | 16.3 | 6.9 | 17.5 | # 可视化工具 ## 使用方法 1. 访问 [VeriGUI.2077ai.org](https://verigui.2077ai.org) 2. 选择对应任务数据文件夹 3. 查看可视化结果 ## 功能特性 - 交互式事件时间线可视化 - 支持多种事件类型（如MOUSE_DRAG、MOUSE_UP、TAB_CHANGE等） - 视频播放同步 - 跳转至指定操作的功能 # 数据集结构 VeriGUI/ ├── task_001/ │ ├── data.json # 完整任务标注 │ └── video.mp4 # 任务执行视频录制 └── task_002/ ├── data.json └── video.mp4 ## 任务结构 json 📋 根任务 ├── instruct (字符串): 主任务指令 ├── result (字符串): 完整任务的最终预期结果 ├── actionLength (整数): 总高级步骤数 └── actions (数组): 逐步骤操作列表 │ └── 📝 步骤对象 ├── checked (布尔值): 该步骤是否已验证 ├── instruct (字符串): 该步骤的子任务指令 ├── result (字符串): 该步骤的预期结果 └── innerActions (数组): 该步骤内的底层GUI操作列表 │ └── 🖱️ 操作对象 ├── type (字符串): GUI操作类型 ├── url (字符串): 当前网页URL ├── rawHtml (字符串): 原始HTML内容（可选） ├── time (整数): 时间戳（毫秒） ├── _delete (布尔值): 是否忽略该操作 └── info (对象): 详细操作信息 ├── clientX/Y (整数): 相对于视口的鼠标坐标 ├── pageX/Y (整数): 相对于页面的鼠标坐标 ├── layerX/Y (整数): 相对于图层的鼠标坐标 ├── screenX/Y (整数): 相对于屏幕的鼠标坐标 ├── offsetX/Y (整数): 相对于目标元素的鼠标坐标 ├── altKey/shiftKey/metaKey (布尔值): 修饰键状态 └── target (对象): 目标DOM元素信息 ├── innerText (字符串): 目标元素文本内容 ├── className (字符串): CSS类名 └── [其他DOM属性] # 待办事项 ## 📊 数据集扩展 - [ ] **桌面环境数据采集** - [ ] 办公软件交互（如Microsoft Office、LibreOffice等） - [ ] 专业工具（如Adobe Creative Suite、集成开发环境等） - [ ] **身份验证与用户管理任务** - [ ] 包含表单验证的用户注册流程 - [ ] 跨平台登录流程 - [ ] 多因素认证（2FA/MFA）处理 - [ ] 通过邮件/短信进行账户验证 - [ ] 验证码与验证码交互 - [ ] 将现有130个任务扩展至**500个以上** - [ ] 保持各分类的均衡分布 - [ ] 新增更多跨应用工作流 ## 📈 交互式数据任务 - [ ] 交互式仪表盘导航与数据筛选 - [ ] 图表缩放、平移与提示框信息提取 - [ ] 通过UI控件开展多维度数据探索 - [ ] 通过网页界面进行科研数据库查询 - [ ] 统计分析工具交互 ## 🔧 评估与基准测试 - [ ] **全维度模型性能分析** - [ ] **高级评估指标** # 引用 若您在研究中使用VeriGUI，请引用如下文献： @article{verigui2025, title={VeriGUI: Verifiable Long-Chain GUI Dataset}, author={Shunyu Liu, Minghao Liu, Huichi Zhou, Zhenyu Cui, Yang Zhou, Yuhao Zhou, Wendong Fan, Ge Zhang, Jiajun Shi, Weihao Xuan, Jiaxing Huang, Shuang Luo, Fang Wu, Heli Qi, Qingcheng Zeng, Ziqi Ren, Jialiang Gao, Jindi Lv, Junjie Wang, Aosong Feng, Heng Zhou, Wangchunshu Zhou, Zhenfei Yin, Wenlong Zhang, Guohao Li, Wenhao Yu, Irene Li, Lei Ma, Lei Bai, Qunshu Lin, Mingli Song, Dacheng Tao}, journal={arXiv preprint arXiv:2508.04026}, year={2025} } # 许可证 本数据集采用Apache-2.0许可证发布。
