SPA-BENCH
收藏资源简介:
SPA-BENCH是由华为诺亚方舟实验室和哈尔滨工业大学(深圳)联合创建的智能手机代理评估基准数据集。该数据集包含340个任务,涵盖系统应用和第三方应用,支持中英文双语,旨在评估基于大型语言模型(LLM)和多模态大型语言模型(MLLM)的智能手机代理在真实交互环境中的表现。数据集创建过程中,通过人工标注和验证确保任务质量,涵盖单应用和跨应用任务,难度级别从简单到复杂。SPA-BENCH的应用领域主要集中在智能手机控制和自动化任务,旨在解决智能手机代理在实际应用中的性能评估问题。
SPA-BENCH is a benchmark dataset for smartphone agent evaluation, jointly developed by Huawei Noah's Ark Lab and Harbin Institute of Technology (Shenzhen). It comprises 340 tasks covering both system applications and third-party applications, supports both Chinese and English, and its core objective is to evaluate the performance of smartphone agents powered by Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) in real-world interactive environments. During the dataset construction, manual annotation and validation were implemented to guarantee task quality, with tasks spanning both single-application and cross-application workflows, and difficulty levels ranging from simple to complex. The application scenarios of SPA-BENCH primarily focus on smartphone control and automation tasks, aiming to address the performance evaluation challenges of smartphone agents in real-world practical applications.

- 1SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation华为诺亚方舟实验室, 哈尔滨工业大学(深圳), 伦敦大学学院 · 2024年



