微调 Whisper 帮失语者重建语音：LoRA 让 WER 从 49% 降到 17%
DIY Customized Speech Recognition for a Person with Speech Disabilities
一位脑瘫研究者微调 Whisper 给自己做了实时语音翻译器 AVoz，词错误率从 30% 压到 12%，全程本地运行。代码和训练协议全开源，几小时录音加几天 GPU 时间就能复现，做无障碍应用的开发者别错过。
一篇 arXiv 论文报告了为一位脑瘫合并构音障碍研究者定制的语音识别系统。作者用 LoRA 在消费级硬件上微调 OpenAI 的 Whisper medium 和 large-v3，使用自己录制的语音样本。在 100 条未见过的测试语句上，medium 的词错误率（WER）从 49% 降到 17%，large-v3 从 30% 降到 12%；字符错误率（CER）分别从 26% 降到 7%、从 16% 降到 5%，剩余错误主要是轻微音素替换。
在此基础上，作者开发了 AVoz 移动应用，把他本人的语音实时转成合成语音播放给听者，全部处理在本地完成，音频不出设备。论文还公开了完整的 LoRA 配置、实现代码和逐步录音训练协议，其他言语障碍者或其家属只需几小时标注音频和几天的 GPU 时间即可复现整套流程。
通用 ASR 系统对中重度言语障碍者的识别失败是长期痛点，这项工作用个人化微调给出了可复制的低成本方案，对无障碍技术和开发者社区都有直接参考价值。
DIY Customized Speech Recognition for a Person with Speech Disabilities
General-purpose state-of-the-art automatic speech recognition (ASR) systems fail systematically for speakers with moderate to severe speech disabilities. This paper describes the design, implementation, and evaluation of a customized ASR system built for the first author, a researcher with cerebral palsy and dysarthria. We fine-tuned OpenAI's Whisper medium and large-v3 models using Low-Rank Adaptation (LoRA) on audio samples recorded from the first author's own speech on consumer-grade hardware. On a held-out test set of 100 previously unseen utterances, the corresponding transcriptions resulted in a Word Error Rate drop from 49% to 17% for medium and from 30% to 12% for large-v3, while the Character Error Rate fell from 26% to 7% for medium and from 16% to 5% for large-v3, indicating that the residual errors are minor phonemic substitutions. Based on these models, the first author developed AVoz, a mobile application that acts as a real-time interpreter, converting his speech into synthesized voice for the listener. All processing is performed locally, ensuring that no audio data leaves the user's machine. We argue that the methodology is replicable. Moreover, we publish the full LoRA configuration, the implementation, and a step-by-step recording and training protocol, so that other people with speech disabilities, or their family members and caregivers, can build their own systems with a few hours of labeled audio data and a couple of days of GPU time.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Whisper, LoRA, 语音识别, 无障碍.
Primary source verification: published directly by arXiv: OpenAI with first-party authority.
