长时运行智能体终于有了解决上下文膨胀问题的思路——做Agent或长链推理的开发者值得关注，它可能改变你处理长期记忆的方式。
研究发现，长时间运行的语言智能体如果定期暂停并整合记忆，性能会更好。当前Transformer模型随着上下文增长，注意力机制需要检查更多历史token，导致推理变慢且成本增加。论文提出在模型中引入“睡眠阶段”：暂停推理，多次重读近期上下文，将有用信息写入固定大小的记忆层，然后清空短期注意力缓存。这样，模型在睡眠时进行额外计算，而正常推理仍保持单次前向传播的高效。实验表明，睡眠时间越长，模型在需要深度推理的复杂任务上表现越好，尤其当旧信息已不在注意力缓存中时。
Long-running language agents may work better if th…
Long-running language agents may work better if they periodically stop to consolidate memory.
The problem is that today’s transformer agents get slower and more expensive as their context grows, because attention has to keep checking more past tokens.
The usual fix for long context is to keep more tokens nearby, but that turns every next-token prediction into a larger search through the past.
The sharper idea here is that memory is not only storage.
Sometimes the hard part is converting a messy stretch of experience into a state that can actually be used later.
So the paper’s idea is to add a sleep phase, where the model pauses, rereads recent context several times, writes the useful information into fixed-size memory layers, and then clears the short-term attention cache.
During sleep, the model runs several offline passes over recent context, writes the result into fast weights inside its state-space blocks, then clears the attention cache.
This means the model pays extra compute while sleeping, not while answering, so normal prediction can still happen with 1 forward pass.
The authors test this on cellular automata, graph lookup, and GSM-Infinite math problems, where the model must use old information that is no longer sitting in its attention cache.
The main result is that longer sleep improves performance, especially on harder cases that need deeper reasoning rather than just remembering a fact.
The big deal is that long-horizon agents may not need to carry bigger and bigger raw context forever, because they can consolidate the important parts and safely forget the raw tokens.
----
Link – arxiv. org/abs/2605.26099
Title: "Language Models Need Sleep"