论文72°

PolySpeech-100:覆盖110种语言方言的语音理解基准

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

精选理由

语音理解基准终于覆盖了方言和低资源语言,做多语言语音模型或方言应用的团队可以直接用这个基准来评估自己的模型,避免只测英语的偏科问题。

AI 摘要

PolySpeech-100 是一个大规模语音理解基准,覆盖 110 种语言变体,包括 19 种中文方言和 80 多种低资源语言。它采用混合构建流程,结合人工录音和指令驱动合成语音,解决了现有基准偏重高资源语言、仅关注 ASR 而非语义推理、忽视方言的问题。评估 22 个模型(如 Gemini-3、GPT-Audio、Qwen2.5-Omni)后发现:开源端到端模型在重方言上优于级联系统,但低资源语言上性能严重下降;链式思维提示在零样本设置下反而降低大多数模型的语音理解能力。该基准为下一代包容性语音大模型建立了严格标准,数据和代码已开源。

原文 · arXiv cs.AI

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription. Existing benchmarks suffer from three critical limitations: a pronounced bias towards high-resource languages, a focus on low-level recognition (ASR) rather than semantic reasoning, and a neglect of regional dialects. To bridge this gap, we introduce PolySpeech-100, a massive-scale benchmark designed to assess `native-level' speech comprehension across 110 linguistic variants. We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech, allowing us to cover 19 distinct Chinese dialects and over 80 low-resource languages. Extensive evaluation of 22 state-of-the-art models (including Gemini-3, GPT-Audio, and Qwen2.5-Omni) yields pivotal insights. First, we demonstrate that open-source E2E models outperform Cascade (ASR+LLM) systems on heavy dialects, proving that direct audio processing preserves critical paralinguistic cues and prosodic features (e.g., intonation, stress) that are often lost in standard transcription. Second, we reveal a significant performance gap: while commercial models maintain robustness, open-source models suffer catastrophic degradation on low-resource languages. Finally, counter-intuitively, we observe that under standard zero-shot settings, Chain-of-Thought prompting frequently degrades speech understanding performance for most evaluated models, revealing a potential modality alignment gap in current architectures. PolySpeech-100 establishes a rigorous standard for the next generation of inclusive, omni-capable Speech-LLMs. The data, demo, and code are publicly available at https://github.com/YoungSeng/PolySpeech-100.