Beyond Word Error Rate：流式听写 ASR 的失败分类研究
Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR
这篇论文把 11 个 ASR 系统的错误按四个层级逐条分类，发现 WER 相近的产品在耳语场景下表现天差地别。如果你在做语音输入或听写功能，Silero VAD 会把 Whisper 的耳语 WER 从 10.1% 推到 51.4% 这个结论直接关系到你的技术选型，全部数据集也开源了。
arXiv 论文提出超越词错误率（WER）的 ASR 评估方法，将 106 条英文听写音频（含安静环境、开风扇、耳语三种条件）流式送入 11 个 ASR 系统，并把每个错误按识别、输出格式、语言处理、音频丢弃四个层级分类。结果显示 WER 相近的系统失败方式差异很大：Zoom Scribe 与 GPT-Live-Transcribe 在安静英语上的 WER 分别为 7.4% 和 7.3%，但耳语英语下急剧恶化到 34.7% 和 8.8%。
技术细节上，Zoom 与两个 AssemblyAI 模型在耳语音频中漏掉 26-28% 的词，即使关闭端点检测仍漏 21-26%，而其他七个系统漏词不超过 3%；在 107 位 EARS 说话人上验证，同样的四个系统耳语漏词 7-31%。语言处理方面，Gemini、Deepgram 与自托管 Whisper 在仅选英语时会丢弃 86-99% 双语录音中的非英语部分。另外，Whisper 前置 Silero 语音活动检测器后，耳语 WER 从 10.1% 升至 51.4%。
对开发者而言，该研究说明部署流式听写时不能只看 WER 排名，VAD 前置与端点检测策略对低音量语音的影响可能超过模型本身的选择。代码、录音、参考文本与全部转写已公开。
Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR
Speech recognition (ASR) systems are usually compared by word error rate (WER) on recorded files, after punctuation, capital letters and number format are removed. In dictation, the audio is streamed live and the text goes into the user's message as written, so what matters is which mistakes the user has to fix. The test set, streamed to eleven ASR systems, comprises 106 scripted English dictations from one speaker (read in a quiet room, with a fan on, and whispered), 20 clips in each of eleven other language and dialect sets, and 400 recordings that join clips in two languages. Every error was sorted by a fixed rule into four levels: recognition, output formatting, language handling and, for the self-hosted models, audio dropped before the model. Systems with similar WER failed in different ways. Zoom Scribe and OpenAI's GPT-Live-Transcribe have almost the same WER on quiet English (7.4% and 7.3%), but 34.7% and 8.8% on whispered English. Zoom and both AssemblyAI models left out 26-28% of whispered words, and still 21-26% with their end-of-turn detection relaxed or off; seven other systems left out at most 3%. Qwen3-ASR's 32% came from the speech detection in our own deployment (6% when relaxed). Across 107 EARS speakers, the same four systems lost 7-31% of whispered words, while six others lost under 2%. With only English selected, Gemini, Deepgram and our self-hosted Whisper lost the non-English part of a two-language recording in 86-99% of recordings, against 33-46% with both languages selected, while GPT-Live-Transcribe, OpenAI's realtime Whisper and Meta lost it in at most 20%. Running Whisper behind a Silero voice activity detector that drops audio judged to be silence raised whispered WER from 10.1% to 51.4%. Most English results come from one read speaker. The code, the recordings, both references and every transcript are released.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include ASR, Whisper, Silero VAD, 语音识别.
Primary source verification: published directly by arXiv: OpenAI with first-party authority. Cross-source corroboration: 4 independent sources have covered this development (including ElevenLabs, Gorden Sun, AWS Machine Learning Blog), confirming industry-wide signal strength.
