论文

MWE-ECL 诊断基准:可检索的远距离上下文未必能改变模型的局部词 prior

MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors

精选理由

这篇论文造了个诊断集,专门测长上下文模型“找到了但没用上”的问题,DeepSeek 两个模型都被测出检索满分但解读掉分,做长文本评测的可以看看。

MWE-ECL 是一个双语诊断基准,测试远距离话语锚点被明确检索到之后,能否改变模型对多词表达的局部偏好解读。在 0-128K 网格的八个英文测试面板上,prior 冲突项的检索对照准确率达 0.989-1.000,但冲突覆盖率为 0.806-1.000,prior 正确决策的保留率则为 0.977-1.000。DeepSeek V4 Pro 在同一调用内检索准确率为 1.000,解读仅为 0.900-0.920,说明检索与解读的差距并非只因分开调用。DeepSeek V4 Flash 在 512K 和 1M 下检索保持完美,但解读下降,且 foil 线索对 no-anchor prior 的影响大于对检索的影响。

原文 · arXiv: DeepSeek

MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors

Long-context evaluations often test whether a model can recover distant evidence, but recoverability does not guarantee behavioral influence. We test the prediction that a distant discourse anchor can remain explicitly recoverable yet fail to change the locally preferred reading of a familiar multiword expression; such failures should concentrate when the model's no-anchor default conflicts with the anchor, while prior-correct decisions remain largely preserved. We introduce Multiword Expression Effective Context Length (MWE-ECL), a bilingual diagnostic whose matched anchor-retrieval, no-anchor prior, and interpretation prompts measure explicit recoverability, model-observed defaults, and anchor-conditioned decisions, respectively. Across eight English deployment panels on a shared 0-128K grid, retrieval-control accuracy on prior-conflict items is 0.989-1.000, prior-conflict override spans 0.806-1.000 (0.809-1.000 after conditioning on correct retrieval), and preservation of prior-correct decisions remains 0.977-1.000. A same-call control querying retrieval and interpretation in one prompt reproduces the gap for DeepSeek V4 Pro (1.000 retrieval versus 0.900-0.920 interpretation), showing that separate invocations are not its sole explanation; smaller or absent gaps in the other two models bound its generality. For DeepSeek V4 Flash, separate prompt-fit tests retain perfect retrieval with lower interpretation at 512K and 1M, while foil-consistent cues shift the no-anchor prior far more than retrieval; cross-model cue effects are heterogeneous. A separately reported 10-family Chinese subset shows similar descriptive gaps, but imperfect retrieval for some models prevents an integration-only attribution. MWE-ECL therefore evaluates whether explicitly recoverable distant context changes a competing local semantic decision.