Google 论文提出 RRSI：防止智能体 harness 自动进化过拟合基准
Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's ha...
Google 这篇论文值得做 Agent 的人细读：自动优化 harness 会让你 eval 分数涨但真实任务变差，RRSI 用双向正则化解决了这个问题。
Google 的论文研究智能体 harness 自动进化中的过拟合问题：Meta-Harness 在 Harvey LAB evolve split 上达到 93.0，但在 JobBench、GDPval 和 APEX-Agents 三个分布外基准上仅提升 0.3 到 1.5 分。RRSI 在提议者侧加入随时间收缩的编辑预算，在评估侧用 critic 拒绝基准特定编辑、pruner 清理代价过高或无用的编辑。RRSI 在 evolve split 上得 90.5，但在三个 held-out 基准上提升 3.5 到 4.7 分，同时把每轮 token 消耗从 3.80M 降到 2.42M。配合 Gemini 3.5 Flash，RRSI 将 Terminal-Bench 2.1 成绩从 64.6 提到 78.7，并将 2.2 分的提升迁移到 SWE-bench Verified。
Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's ha...
Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's harness, your eval score can go up while the agent gets worse on real tasks. This paper shows how to prevent that. Of five harness-evolution methods compared on agentic workspace tasks, RRSI scored the lowest on the tasks it evolved against and highest on all three out-of-distribution benchmarks. Automated harness evolution proposes edits to prompts, control flow, tools and memory, keeps the ones that raise the score, and repeats. The authors show this overfits the training tasks. Meta-Harness reached 93.0 on the Harvey LAB evolve split but gained only 0.3 to 1.5 points on JobBench, GDPval and APEX-Agents. RRSI adds regularization on both sides of the loop. The proposer gets an edit budget that shrinks over time and is pushed toward directions it has not tried. A critic rejects benchmark-specific edits, and a pruner removes edits that are too small, too costly or no longer useful. RRSI scored 90.5 on the evolve split and gained 3.5 to 4.7 points on the three held-out benchmarks. In the ablation, unregularized evolution used 3.80M tokens per trial against 2.42M for RRSI. With Gemini 3.5 Flash, RRSI raised Terminal-Bench 2.1 from 64.6 to 78.7 and carried a 2.2-point gain over to SWE-bench Verified. Paper: arxiv.org/abs/2609.24972 Chat with Paper: academy.dair.ai/papers/rrsi-re… 💬 3 🔄 5 ❤️ 12 👀 1424 📊 7 ⚡
