研究评估英语-约鲁巴语代码切换语音的ASR和音频语言模型
Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech
这个研究很实用，它详细分析了不同AI模型在处理英语和约鲁巴语混合语音时的表现差异，特别是那些切换点错误率的数据，对做相关研究或评估模型的人很有参考价值。
本文对11个现代系统（6个ASR模型和5个音频语言模型）在英语-约鲁巴语代码切换语音上进行了切换感知评估，使用2000句的评估集和共享评分管道。除了词错误率（WER），还报告了切换位置诊断指标：切换入口标记错误率（SETER）、窗口化切换点错误率、语言特定错误率和非重音符号敏感的WER。研究发现，整体WER隐藏了代码切换行为。WER最高的系统（一个ASR模型）在WER上与领先的音频语言模型没有显著区别，但音频语言模型在所有切换感知指标上都表现更好。在忠实系统中，约鲁巴语标记识别几乎完全崩溃（错误率0.97），而英语标记识别效果更好，错误集中在进入约鲁巴语的切换点。几个生成式音频语言模型作为精确转录器失败，产生翻译、冗余和提示词泄露，这些行为高度依赖提示词。
Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech
Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavior on code switched speech in low resource, diacritic rich languages remains poorly characterized. We present a switch aware evaluation of eleven modern systems (six ASR models and five audio LMs) on English Yoruba code-switched speech, using a deterministic 2000 utterance evaluation set and a shared scoring pipeline. Beyond word error rate (WER), we report switch localized diagnostics: a switch entry token error rate (SETER), windowed switch point error rates, language specific error rates, and a diacritic insensitive WER. Our central finding is that aggregate WER hides code switching behavior. The best system by WER (an ASR model) is statistically indistinguishable from a leading audio LM on WER, yet the audio LM is significantly better on every switch localized metric. Across faithful systems, Yoruba token recognition collapses (error 0.97 for almost all systems) while English tokens are recognized far better, and errors concentrate sharply at switches into Yoruba. Several generative audio LMs fail as exact transcribers, producing translation, verbosity, and prompt leakage that are strongly prompt dependent. We release manifests, metric implementations, and evaluation scripts to support reproducible, switch aware benchmarking for African code switched speech.
