MIST基准与SCOPE方法:让模型学会选择性信任上下文

Learning When to Trust via Selective Context Preference Optimization

精选理由

这篇论文把“信不信上下文”当成独立问题来训练,用MIST测出模型容易被一条错误信息带偏,SCOPE能减少这种翻车还不影响正常表现。

AI 摘要

论文提出MIST基准,将每道推理题置于干净、误导、正确上下文、无关上下文四种条件下。配套SC2W指标衡量误导信号使原本正确答案变错的频率。作者发现该脆弱性在多种开源模型上普遍存在。提出的SCOPE方法通过匹配偏好对优化DPO目标,在保持干净上下文准确率的同时显著降低SC2W。

原文 · arXiv cs.LG

Learning When to Trust via Selective Context Preference Optimization

Language models increasingly condition their answers on external signals, and a single misleading one can turn a correct answer wrong. The obvious remedy, training models to resist such signals, hides a failure mode: a model that ignores all context looks robust yet is useless when the context is worth trusting. We recast the problem as selective trust and introduce MIST, a human-annotated benchmark that renders each reasoning item under four matched conditions (clean, misleading, correct-context, and irrelevant-context), together with SC2W, a paired metric counting how often a misleading signal flips a clean-correct answer to wrong. Across a comprehensive benchmark study, we observe that such a susceptibility is universal. We then propose SCOPE, which mines clean-correct/misleading-wrong failures and optimizes a standard Direct Preference Optimization (DPO) objective over matched preference pairs balanced equally across all four conditions, rather than over misleading items alone. Our approach substantially reduces SC2W on popular open-sourced models while preserving accuracy when the added context is clean, correct, or irrelevant. With this work, we argue that models should be judged on selective trust, not on resistance alone.