这篇论文揭示了自我改进智能体的脆弱性，指出任务顺序和规范不足会影响其性能，对AI安全研究有重要启示。
研究人员对两种基于记忆的自我改进智能体方法进行了重新评估，通过多次运行量化方差并随机打乱任务顺序。实验表明，智能体评估在复杂环境和多步骤任务中存在固有噪声，自我改进循环会进一步放大这种噪声。研究发现智能体的改进高度依赖于任务顺序，先前研究采用的默认排序构成了隐式课程，成为成功的隐藏前提。通过分析智能体的记忆，研究者假设任务和环境规范不足是导致脆弱性的原因，并通过添加更详细的信息部分验证了这一假设。
On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification
Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple runs to quantify variance, and (2) randomly shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.