研究揭示医疗视觉模型按平均准确率评测会掩盖单一病症的完全失效
Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models
这篇审计会改变你看医疗模型榜单的方式：两个知名开源模型对气胸全部漏诊，还能流畅描述体征再否认它。做医疗 AI 选型或部署的人必须看，评估任何 VLM 都该按病症拆分指标，别信平均分。
一篇新论文审计了五个开源医疗视觉语言模型：MedGemma-4B、Lingshu-7B、HuatuoGPT-Vision-7B、Qwen2.5-VL-7B 和 MedVLM-R1，使用按性别、年龄、病症和标签平衡的 6048 道 ChestX-ray14 问题。在 432 张气胸阳性 X 光片上，HuatuoGPT-Vision 和 Qwen2.5-VL 检出 0 例，而 MedGemma 和 Lingshu 分别检出 38% 和 67%。两个失效模型对全部 864 道气胸问题一律回答"否"，平衡准确率只有 0.500，等于随机猜测。
失效原因排除了提示方式、图像分辨率、标签歧义和模型规模：同一模型在受控对比下对结节召回率 79%、对气胸只有 6%；放大十倍到 72B 参数后 Qwen2.5-VL 仍只检出 4%；在带放射科医生标注掩膜的独立数据集 SIIM-ACR 上，失效模型对 432 例确诊气胸（含 144 例最大病灶）的检出仍为零。更危险的是，这些模型会先流畅描述气胸的解剖学特征、随后否认它存在，且不作任何保留或拒答。
通过读取模型的 logits 和内部状态，作者证明这不是"看不见"：线性探针能从隐藏表示中恢复气胸信息，说明病症在内部已被表征，只是在输出层被抑制。论文结论是平均准确率无法证明医疗 VLM 能识别某个病症，必须报告按病症拆分的召回率和特异度。
Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models
Medical vision-language models (VLMs) are compared mainly by accuracy averaged across findings, a summary that can hide complete failure on a single diagnosis. We audit five open medical VLMs (MedGemma-4B, Lingshu-7B, HuatuoGPT-Vision-7B, Qwen2.5-VL-7B, and MedVLM-R1) on 6,048 ChestX-ray14 questions balanced by sex, age, finding, and label. On the 432 pneumothorax-positive X-rays, HuatuoGPT-Vision and Qwen2.5-VL detect zero cases while MedGemma and Lingshu detect 38% and 67% of the same images. The two failing models answer "no" to all 864 pneumothorax cases (balanced accuracy 0.500, chance), yet produce fluent, anatomically specific explanations that name the signs of a collapsed lung and then deny them, with no hedge or abstention. This failure is not an artifact of prompting, image resolution, label ambiguity, or model scale. It persists across four prompt formats, at native resolution, and on single-label cases. A within-model control shows the same forced prompt gives HuatuoGPT 79% recall on nodule but only 6% on pneumothorax. It survives a tenfold scale increase (Qwen2.5-VL-72B detects 4%) and replicates on SIIM-ACR, an independent dataset with radiologist-drawn masks, where the failing models detect 0 of 432 confirmed pneumothoraces, including 0 of the 144 largest lesions. Reading the models' logits and internal states shows this is not blindness. The two failing models are highly confident yet no better than chance and a linear probe recovers pneumothorax from their hidden representations even where recall is zero, so the finding is represented internally but suppressed at the output. Aggregate accuracy therefore cannot establish that a medical VLM can see a given finding, because a model that misses one can still describe it convincingly. Reporting per-finding recall and specificity on balanced cases is needed to tell the difference.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include MedGemma, HuatuoGPT, Qwen2.5-VL, ChestX-ray14.
Source attribution: curated from arXiv cs.LG (@Shreya Ajay Kale, Angad Ripudaman Singh Bajwa).
