GAIR/AgencyBench

AgencyBench是一个全面的基准测试，源自日常AI使用，评估了6种核心代理能力在32个真实世界场景中的表现，包含138个具有特定查询、交付物和评分标准的任务。这些场景平均需要90次工具调用、1百万令牌和数小时的执行时间来解决。数据集支持自动化评估，通过用户模拟代理提供迭代反馈，并使用Docker沙箱进行视觉和功能性的基于标准的评估。

AgencyBench is a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics. These scenarios require an average of 90 tool calls, 1 million tokens, and hours of execution time to resolve. The benchmark enables automated evaluation by employing a user simulation agent for iterative feedback and a Docker sandbox for visual and functional rubric-based assessment.