用从未发生的对话训练ASR:合成对话提升低资源语言识别

Efficient ASR Training with Conversations that Never Happened

精选理由

低资源语言ASR团队终于有了可落地的数据增强方案——用LLM+TTS生成对话数据,效果堪比数倍真实数据。做多说话人语音识别的开发者值得一试,尤其适合匈牙利语等小语种场景。

AI 摘要

低资源语言和垂直领域的对话式语音识别受限于多说话人训练数据稀缺。研究者提出一种数据增强流水线:先由LLM生成带参与者元数据的场景级对话,再将说话人属性映射到TTS语音配置,最后组装成说话人感知的合成对话。在匈牙利语BEA-Dialogue基准上,仅用67小时真实对话加636小时合成数据训练的模型,性能超过用2700小时真实语音训练的零样本模型。该方法适用于任何语言,且LLM生成器选择和合成数据组成对效果影响显著。

原文 · arXiv cs.AI

Efficient ASR Training with Conversations that Never Happened

Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata, maps speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations. We evaluated five LLM families under single-generator, fixed-budget mixture, and scale-up settings using the same FastConformer-Large training recipe for each one. We ran comprehensive evaluations on the Hungarian BEA-Dialogue benchmark corpus, with the method itself being applicable to any language given the resources for each component. The results show that synthetic conversations consistently improve speech recognition performance, but generator choice and data composition strongly affect the gains. Our largest training configuration, using only 67 hours of real conversations and 636 hours of simulated data, achieves better performance on the evaluation benchmark than a zero-shot model trained on 2700 hours of Hungarian speech. These findings indicate that LLM-generated conversational data synthesized with TTS is a practical complement to real conversational corpora for speech model training.