新指标 Recall Fidelity Score 量化 LLM 记忆失真如何影响预测能力
Approximate Recall, Approximate Forecasts: Recall as a Diagnostic for LLM Forecasting Errors
这篇论文把 LLM 预测误差的来源拆到了记忆保真度上，还给出了可复现的 Recall Fidelity Score 指标。做模型评测或搭建预测类智能体的开发者可以拿来当作诊断工具，比单纯看预测市场得分更有解释力。
一篇 arXiv 论文提出 LLM 存在“近似召回”现象：模型能记住过去事件的粗略关联，却丢失具体细节，例如能答对 Anna Sawai 获得第 76 届 Emmy 奖最佳剧情类女主角，却把事件日期从 9 月 15 日错说成 9 月 22 日。作者设计了新指标 Recall Fidelity Score，用来衡量模型内部知识的不精确程度。论文核心问题是这种记忆保真度损失是否会传播到模型对未来事件的预测能力上。
在已结算的预测市场数据上，实验显示：模型对语义相似的历史事件召回保真度越高，在已结算市场上的预测误差越低，同一信号也能预测模型报价与开放市场价格的距离更近。这把预测市场基准评估从只看结果分数，推进到解释误差来源的机制层面。
Approximate Recall, Approximate Forecasts: Recall as a Diagnostic for LLM Forecasting Errors
Large Language Models (LLMs) exhibit approximate recall: they retain coarse associations about past events while losing fidelity on specific details. Past research on LLMs has quantified different models' forecasting ability in real-world events using prediction markets, but has not studied how losing said fidelity impacts their forecasting abilities. For example: while an LLM correctly recalls that the answer to: ''Did Anna Sawai win the Best Drama Actress in the 76th Emmy Awards?'' is ''Yes,'' it incorrectly states the event date as September 22 rather than September 15. In this paper, we introduce a novel metric called the Recall Fidelity Score to study how imprecision in internalized knowledge propagates to forecasting capability on future events. On resolved historical markets, we find that higher local recall fidelity around semantically similar past events predicts lower forecasting error on resolved markets, and the same signal predicts smaller distance from open markets prices.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, 预测市场, Recall Fidelity Score, 评测基准.
Source attribution: curated from arXiv cs.LG (@Prashanth Bhaskara, Shubhaankar Gupta, Seojoon Yeon).
