研究显示 coding agent 持续自我进化会遭遇灾难性遗忘
Can Coding Agents Sustain Continual Evolution without Forgetting?
清华团队的实证研究，发现 agent 进化完一个任务再进化下一个，之前练的技能会大幅退化，做 agent 开发的值得看看这些避坑发现。
清华大学团队研究了 agent harness 在多个任务上的持续自我进化问题，将三个软件工程数据集组织成六个进化方向，测试两种 harness 进化方法。实验发现，用 DeepSeek-V4-Flash-0731 做实验时，Meta-Harness 在 SWE-Bench 上进化后性能达到 94.0%，但继续在 FeatureBench 上进化后，SWE-Bench 成绩在第五轮跌至 32.0%，相对下降 66%。论文基于实验结果总结了应对持续进化中遗忘问题的若干方向，代码已在 GitHub 开源。
Can Coding Agents Sustain Continual Evolution without Forgetting?
Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.
