Papers

论文提出测量语言模型时间数据污染的新方法

Reading or Remembering? Measuring Temporal Contamination in Language Models

Why we picked it

做经济预测或时间敏感评估的人都该看看这篇。它发现只给日期模型就能拿到 0.64 AUC,随机基线会高估模型能力。文中给出的无记忆模型对比方法可以直接搬进自己的评测流程,尤其是金融和宏观研究方向。

一项研究指出,用语言模型解读美联储 FOMC 声明做经济预测时存在时间维度的测试集污染:模型训练语料晚于所读文档,可能只是记住了 2022 年通胀上升等结果。

Body

研究覆盖 2000 至 2026 年的 FOMC 声明与会议纪要,将现代开源模型与仅在固定日期之前文本训练的时间一致模型对比。结果显示,仅凭日期和时代先验,模型就能在无法知晓答案的 GDP 问题上达到 0.

64 的 AUC,说明随机基线并不适用;而与同题目的无记忆模型对比时,多数现代模型能回忆出哪些年份增长强劲。

成员推断得分按时代和文档类型排序,而非按模型是否见过该文档分类,研究用开放预训练语料中的精确计数验证了这一点。在美国 GDP 的实时即时预测中,所有被估计的前瞻溢价都低于宏观样本量可检测的水平,GDP 与通胀上的结果经多重检验校正后无一存活。

研究结论是:前瞻性审计应在相同题目上用无记忆模型直接测量记忆。

Original · arXiv cs.LG

Reading or Remembering? Measuring Temporal Contamination in Language Models

Language models are now used to read central-bank text for economic forecasting, but most were trained on text written after the documents they read. A model reading a 2021 FOMC statement may already know that inflation rose in 2022, so part of its apparent skill can be memory of the outcome, a temporal form of test-set contamination. We measure this memory on FOMC statements and minutes from 2000 to 2026 by comparing modern open models with chronologically consistent models, which are trained only on text from before a fixed date. Asked about past GDP outcomes given only a date, models that cannot know the answer still reach an AUC of up to 0.64 from priors on dates and eras, so chance is the wrong baseline. Against these memory-free models on the same questions, most modern models recall which years had strong growth. Membership-inference scores sort documents by era and document type rather than by whether a model saw them, as exact counts in an open pretraining corpus show. In real-time nowcasts, every lookahead premium we estimate on US GDP is smaller than the forecast comparison can detect at macroeconomic sample sizes, and none on GDP or inflation survives correction for multiple testing. Lookahead audits should therefore measure memory directly against memory-free models on the same items.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include benchmarks, FOMC, GDP forecasts, language models.

Source attribution: curated from arXiv cs.LG (@Arunkumar V, Gangadharan G. R., S. Senthilkumar).