这篇论文说长上下文训练反而会让模型记不住知识，上下文越长越依赖临时找资料。想调长上下文的人建议先看看。
一篇arXiv论文提出信息丰裕悖论，认为训练上下文中相关信息过多会降低模型将其参数化编码的动机。预训练阶段扩大上下文窗口虽能提升语言建模、自然语言理解和closed-book MCQA，但只在中间最优值内有效，之后持续下降。监督微调时，更多任务相关上下文虽能提升带支持场景的表现，却削弱了测试时缺少或误导上下文时的鲁棒性。机制分析显示，长上下文将梯度压力从前馈网络转向注意力模块，导致推理时更依赖上下文。结论表明，追求无限上下文并非简单堆数据。
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that training on longer contexts will only help the model by exposing it to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization. We propose the Information Abundance Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically, thereby increasing reliance on context. In pretraining with long documents, increasing the context window improves language modeling, natural language understanding, and closed-book MCQA only up to an intermediate optimum, after which performance consistently declines. In supervised fine-tuning, more task-relevant train-time context improves performance with supporting context, but reduces robustness when context is absent or misleading at test time. Our analysis suggests that this behavior arises when longer context provides a lower complexity solution. Mechanistically, training with informative context shifts gradient pressure from feed-forward networks, often linked to parametric knowledge, toward attention modules, and causal interventions show that this shift increases reliance on context during inference. Overall, these findings support the Information Abundance Paradox and suggest that scaling toward near-infinite context is not simply a matter of supplying more data, even when high-quality long-context data is abundant.