做语音智能体或对话系统的团队终于有了一个能同时测准确性和体验感的基准——EVA-Bench 覆盖了企业场景和噪声鲁棒性，直接帮你对比不同架构的优劣，建议点开看看具体指标设计。
EVA-Bench 是一个全新的端到端评估框架，专门用于测试语音智能体（Voice Agents）在真实对话场景中的表现。它解决了现有基准无法同时模拟动态对话和全面衡量语音特有失败模式的问题。框架包含 213 个企业级场景，并引入两个复合指标：EVA-A（准确性）和 EVA-X（体验），分别评估任务完成度、忠实度、语音保真度以及对话流畅性、简洁性和轮次时机。在 12 个系统上的测试显示，没有系统能同时在两个指标上超过 0.5，且峰值性能与可靠性能差距显著。该框架已开源，为语音智能体的标准化评估提供了新工具。
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to different agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k - pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license.