这篇论文提出了一个评估医疗思维链是否忠实的新方法,对于关注医疗AI的读者来说,这是一个重要的研究进展。
研究提出医疗思维链扰动审计方法,对14个LLM在四个医疗问答基准上的测试显示,72.9%的编辑不影响答案,链式腐败不影响准确性,去除CoT提示不降低准确性。临床医生重新标注197个扰动问题,98.5%保持原答案。框架和CDR提供评估医疗思维链是否忠实或仅是文档的标准。
Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought
Clinicians read chain-of-thought (CoT) rationales as evidence of medical reasoning, but whether the visible chain plays that role is rarely tested. General-domain CoT-faithfulness probes ignore clinical cost, and medical LLM evaluations treat the chain as a black box. We close this gap with a medical perturbation audit: a 30-operator battery edits both the chain and the question with clinically motivated operators (severity reversal, negation flip, demographic swap, evidence ablation), paired with a chain-update times answer-flip joint analysis that classifies each model by its failure mode. Applied to 14 LLMs on four medical QA benchmarks, three independent tests converge: the Chain-Decoupling Rate (CDR; chain does not register the edit and the answer does not flip) is 72.9% panel-wide on clinically meaningful destructive edits, chain corruption leaves accuracy unchanged, and removing CoT prompting does not reduce accuracy. Two board-certified clinicians re-annotate N=197 perturbed questions; 98.5% leave the gold defensible. The pattern holds across medical and reasoning fine-tuning and scale; on the closed-source tier, where the chain text is unavailable, the answer-side signals are consistent with the same decoupling. Our framework and CDR provide a reusable yardstick for auditing whether medical CoT is faithful or merely documentation.