ROUNDS 框架测试多轮对话诊断:LLM 在无选项场景准确率降至 57.0%
ROUNDS: Source-Framed Diagnostic Revision after Multi-Turn Clinical Evidence
这篇论文做了个挺扎心的实验:让模型在多轮问诊后下诊断,结果没有选项时准确率从 76.9% 掉到 57.0%,而且一句“医生说的”错误意见就能让它推翻自己的正确判断。做医疗 AI 产品或评估的人应该看看这套测试设计。
arXiv 论文提出 ROUNDS 评估框架,在多轮临床证据输入后测试 LLM 的诊断能力。该框架测试了七个模型、来自五个数据集的 1,035 个病例,结果显示:有答案选项时平均准确率为 76.9%,去掉选项后降至 57.0%,两者相差约 20 个百分点。
ROUNDS 的关键设计包括:对比有无答案选项的诊断表现、将同一错误诊断分别以临床医生口吻和外行口吻呈现给模型,以及用五人临床医生小组的书面意见测试模型对共识的敏感度。结果显示,临床医生口吻的误导建议在所有七个模型中都造成比外行口吻更大的准确率下降,差距为 4.6 到 38.2 个百分点。
一个重要发现是:初始诊断正确的模型在收到误导意见后,更倾向于直接改成错误答案而非弃权,许多模型直接采纳了建议中的疾病名称。准确率随支持正确诊断的小组成员数量增加而上升。这暴露了模型在医疗基准高分之外的真实短板——缺少新临床证据时,模型会因外部意见放弃已得出的正确诊断。
ROUNDS: Source-Framed Diagnostic Revision after Multi-Turn Clinical Evidence
Large language models (LLMs) achieve high scores on medical benchmarks, but does this performance translate into reliable diagnosis as a conversation unfolds? In diagnostic conversations, users can provide clinical findings across multiple turns, ask for a diagnosis without supplying answer options, and introduce opinions from others. Models must therefore form diagnoses from the available evidence and decide how to respond to subsequent advice. We introduce ROUNDS, a framework that evaluates these abilities after a fixed sequence of clinical evidence delivered over multiple turns. ROUNDS compares diagnosis with and without answer options, presents the same incorrect diagnosis under clinician and lay framing, and varies support for competing diagnoses in written five-clinician panels. Across seven models and 1,035 cases from five datasets, average accuracy was 76.9% with answer options and 57.0% without them. Clinician-framed misleading advice produced lower final accuracy than lay-framed advice in every model, with differences of 4.6 to 38.2 percentage points. Initially correct diagnoses more often became incorrect answers than abstentions, and many adopted the suggested disease. Conversely, accuracy increased as more panel members supported the correct diagnosis, including when the majority opinion remained unchanged. These findings reveal a gap between medical benchmark performance and reliable diagnosis in conversation: models respond to external opinions, but remain vulnerable to abandoning correct diagnoses without new clinical evidence.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Key entities and technical topics under continuous tracking include ROUNDS, LLM, AI safety, benchmarks.
Source attribution: curated from arXiv cs.AI (@Jinrui Fang, Runhan Chen, Yining Wu, Jiawei Xu, Xu Yang, Jian Yu, Wenqi Shi, Ying Ding, Yuji Zhang).