ServiceNow-AI/eva-bench

EVA-Bench是一个端到端的评估框架，用于对话语音代理，通过组织bot-to-bot音频对话，并在任务准确性和交互体验上评分。它解决了生成真实模拟对话和测量语音特定失败模式质量的核心挑战。该框架包含213个场景，覆盖3个企业语音领域：航空客户服务管理（CSM）、医疗人力资源服务交付（HRSD）和企业IT服务管理（ITSM），专注于那些通过电话与人工代理交流是现实且必要的情况。框架采用复合评分系统，包括EVA-A（准确性）和EVA-X（体验），并支持架构无关的评估，适用于级联、语音到语音和混合语音代理架构。数据集通过合成方法构建，使用OpenAI GPT-5模型生成，并经过多轮验证，确保每个场景都有明确的地面真值和可解性。

EVA-Bench is an end-to-end evaluation framework for conversational voice agents that orchestrates bot-to-bot audio conversations and scores them on both task accuracy and interaction experience. It addresses the two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. This release contains 213 scenarios across 3 enterprise voice domains — airline customer service, healthcare HR, and IT service management — focused on situations where speaking to a human agent over the phone is realistic and necessary. The framework introduces two composite metrics, EVA-A (Accuracy) and EVA-X (Experience), and is architecture-agnostic, applying to cascade, speech-to-speech, and hybrid voice-agent architectures. The dataset is constructed synthetically using OpenAI GPT-5 models with multiple rounds of validation, ensuring each scenario has a clear ground truth and is solvable.