Switch Aware Evaluation of ASR and Audio LMs on English Yoruba Code-Switched Speech
Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech
This study provides a detailed, switch-aware evaluation of how modern ASR and audio language models perform on a specific and challenging code-switched language scenario, which is valuable for anyone working on or evaluating speech recognition systems for multilingual contexts.
This paper presents a new evaluation of eleven modern ASR and audio language models on English Yoruba code-switched speech. It uses a 2000-utterance evaluation set and a shared scoring pipeline. Beyond word error rate (WER), the study reports switch localized diagnostics like switch entry token error rate (SETER) and windowed switch point error rates. The central finding is that aggregate WER hides code-switching behavior. The best system by WER is statistically indistinguishable from a leading audio LM on WER, yet the audio LM is significantly better on every switch-localized metric. Errors concentrate sharply at switches into Yoruba, and several generative audio LMs fail as exact transcribers, producing translation and verbosity.
Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech
Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavior on code switched speech in low resource, diacritic rich languages remains poorly characterized. We present a switch aware evaluation of eleven modern systems (six ASR models and five audio LMs) on English Yoruba code-switched speech, using a deterministic 2000 utterance evaluation set and a shared scoring pipeline. Beyond word error rate (WER), we report switch localized diagnostics: a switch entry token error rate (SETER), windowed switch point error rates, language specific error rates, and a diacritic insensitive WER. Our central finding is that aggregate WER hides code switching behavior. The best system by WER (an ASR model) is statistically indistinguishable from a leading audio LM on WER, yet the audio LM is significantly better on every switch localized metric. Across faithful systems, Yoruba token recognition collapses (error 0.97 for almost all systems) while English tokens are recognized far better, and errors concentrate sharply at switches into Yoruba. Several generative audio LMs fail as exact transcribers, producing translation, verbosity, and prompt leakage that are strongly prompt dependent. We release manifests, metric implementations, and evaluation scripts to support reproducible, switch aware benchmarking for African code switched speech.
