Papers

arXiv 论文:对话历史会让 LLM 推理准确率大幅下降

The Silent Cost of Conversation History in LLM Reasoning

Why we picked it

做多轮 Agent 或对话系统的人该看看这篇。它用 48 个数学任务和 CDR-Stream 基准证明对话历史会把准确率拖垮,摘要替代能高出 0.5 以上,而且常用监控指标根本测不出问题。工程上直接可用的结论:状态放 transcript 外面。

arXiv 论文 2611.20071 研究了多轮对话中累积的历史记录对 LLM 推理准确率的影响。Study 1 在 48 个分片数学任务上发现,保留模型自己的历史回复使 GPT-4o-mini 最终成功率下降 0.21,Mistral-7B 下降 0.29(Holm 校正后 p 值分别为 0.026 和 0.004)。

Body

Study 2 提出 CDR-Stream 基准:100 轮状态追踪对话,每轮有精确答案并做按模型难度校准。在完整历史下,同等难度时准确率从约 0.6 在二十轮内跌到 0.1;而用求解器验证过的状态摘要替代历史,可把准确率保持在高出 0.52-0.59 的水平(p<0.001)。一致性、逻辑连贯性等过程指标的下降远小于准确率,且稳定性反而上升,因此这些指标无法提前预警。

作者测试的一致性指标、配对就绪度测量和校准后的单流漂移检测器都没有给出可靠的跨模型早期预警;单条 100 轮流式数据只能检测到极大的降幅。论文建议系统设计者把任务状态存放在对话记录之外,并跨会话进行监控。

Original · arXiv cs.AI

The Silent Cost of Conversation History in LLM Reasoning

Large language models (LLMs) in multi-turn information systems accumulate their own earlier replies and a growing task state. We asked whether this conversation history lowers accuracy, and whether process-level quality metrics warn before it does. Each analysis was fixed before its outcomes were computed. In Study 1, on 48 sharded math tasks, retaining its replies lowered final success by 0.21 for GPT-4o-mini and 0.29 for Mistral-7B (Holm-adjusted p = 0.026 and 0.004); Llama-4 Scout showed the same direction, not significant. Study 2 introduces CDR-Stream, 100-turn state-tracking dialogues with an exact answer per turn and per-model difficulty calibration. With the full history, accuracy fell from about 0.6 to 0.1 within twenty turns at matched difficulty, whereas a solver-verified state summary kept it 0.52-0.59 higher (Holm-adjusted p less than 0.001). Consistency and logical coherence fell far less than accuracy, and stability rose. None of these metrics, a paired readiness measure or calibrated single-stream drift detectors gave robust, model-general early warning, although in Study 2 the repeat-based metrics were complete only for GPT-4o-mini. In the monitors we tested, a single 100-turn stream had power only for very large drops. Systems built on LLMs should keep task state outside the transcript and monitor across sessions.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GPT-4o-mini, Mistral-7B, CDR-Stream, agents.

Source attribution: curated from arXiv cs.AI (@Ali Şenol).