arXiv 论文：专家身份会让 LLM 改错更多答案而非修正错误
When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
做 Agent 多轮协作的人该看看：模型听到"专家不同意"就乱改答案，5 个模型全没达标，准确率最多掉 16 个点。论文给了套可复用的配对审计指标，直接能拿来测你自己的修订流程。
arXiv 论文提出一种两阶段受控审计方法：固定问题、证据和模型初始答案，只改变修订情境——中性反思、匿名反对意见、或冠以领域专家头衔的反对意见。对二元决策可精确记账：只有当模型推翻错误答案的频率超过推翻正确答案的 a/(1-a) 倍（初始准确率 90% 时为 9 倍）时，修订才能提升准确率。
在 5 个模型、1,989 个医学、科学、法律领域的有证据支撑决策上，没有任何模型在专家反对下达到这一标准，准确率最多下降 16.1 个百分点。专家标签在 5 个模型中的 4 个上增加了推翻次数，但对准确率的影响从下降 5.2 点到上升 2.4 点不等，说明推翻次数无法作为危害的代理指标。在论点完全相同的情况下，专家标签仍使 3 个闭源模型推翻更多答案。
这意味着构建修订流程的团队不能用"推翻计数"或"最终准确率"单独评估修订环节，需要按模型、按协议采用配对条件化指标。
When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
When language models revise a decision after another agent disagrees, neither reversal counts nor final accuracy shows whether they are fixing mistakes or abandoning correct answers. We introduce a controlled two-pass audit that holds the question, the evidence, and the model's initial answer fixed and varies only the revision context: neutral reflection, anonymous disagreement, or disagreement attributed to a domain expert. For binary decisions the accounting is exact: revision raises accuracy only if it is selective enough, reversing wrong answers more than $a/(1-a)$ times as often as correct ones, where $a$ is initial accuracy (nine times at 90%). Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points. The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, so reversal counts are a poor proxy for harm. With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and protocol.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, 专家标签, 决策修订, arXiv论文.
Source attribution: curated from arXiv cs.AI (@Yupeng Tang, Mingfeng Lin).