想测语音助手被用户打断后能不能接好活?IHBench专门看这个,比谁恢复得自然、不错步骤。闭源模型比开源稳太多了。
IHBench评估语音助手在10个企业领域中断后的恢复能力,包含6种中断类型。27个音频语言模型配置来自OpenAI、Google和开源社区。闭源模型在任务完成度上显著优于开源模型,长对话中性能下降慢约3.3倍,且无音频-文本模态差距。人类研究验证了LLM评判的可靠性,交叉分析显示恢复质量是独立能力维度。
IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows
Voice agents deployed in structured workflows (customer service, healthcare scheduling, account management) must handle frequent user interruptions while maintaining progress through multi-step procedures. Existing benchmarks for speech-capable models focus on the timing of interruptions: barge-in detection, endpointing, and turn-taking dynamics. They leave unmeasured what happens after the interruption: does the agent resume the workflow at the correct step? Does it address the user's interjection? Does it avoid re-delivering content the user already heard? We introduce IHBench (Interruption Handling Benchmark), a benchmark that evaluates post-interruption recovery in voice agents executing state-machine-driven workflows across 10 enterprise domains. Six interruption types are injected at controlled points mid-utterance, with per-interruption evaluation rubrics generated alongside the data. Each interruption is scored on two axes: task fulfillment and recovery quality. We evaluate 27 audio-language model configurations from OpenAI, Google, and the open-weight community. Models vary widely, and recovery quality depends strongly on the interruption type. Across our experiments, closed-weight models are consistently more robust to interruptions than open-weight ones: they win far more often on task fulfillment, degrade roughly 3.3x more slowly as conversations grow longer, and show no audio-versus-text modality gap, whereas the open-weight models lose ground on all three. A human study validates the LLM judge against human annotators, and a cross-benchmark analysis against AudioMultiChallenge indicates that recovery quality is a largely distinct capability axis.