Google Paper Shows RRSI Stops Agent Harness Evolution Overfitting
Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's ha...
Optimizing an agent's harness against a benchmark can inflate eval scores while degrading real-world performance, and this paper quantifies that gap with held-out numbers. Builders running automated prompt or harness optimization should add critic and pruning loops like RRSI's to protect out-of-distribution performance. The paper and a chat-with-paper interface are available via arXiv and dair.ai.
A Google paper compares five methods for automatically evolving agent harnesses, where edits to prompts, control flow, tools and memory are kept if they raise eval scores. The paper shows this process overfits: Meta-Harness reached 93.0 on the Harvey LAB evolve split but gained only 0.3 to 1.5 points on out-of-distribution benchmarks JobBench, GDPval and APEX-Agents. The proposed RRSI method adds regularization via a shrinking edit budget, a critic that rejects benchmark-specific edits, and a pruner that removes low-value edits. RRSI scored 90.5 on the evolve split while gaining 3.5 to 4.7 points on the three held-out benchmarks, and used 2.42M tokens per trial versus 3.80M for unregularized evolution. With Gemini 3.5 Flash, RRSI raised Terminal-Bench 2.1 from 64.6 to 78.7 with a 2.2-point transfer gain on SWE-bench Verified.
Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's ha...
Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's harness, your eval score can go up while the agent gets worse on real tasks. This paper shows how to prevent that. Of five harness-evolution methods compared on agentic workspace tasks, RRSI scored the lowest on the tasks it evolved against and highest on all three out-of-distribution benchmarks. Automated harness evolution proposes edits to prompts, control flow, tools and memory, keeps the ones that raise the score, and repeats. The authors show this overfits the training tasks. Meta-Harness reached 93.0 on the Harvey LAB evolve split but gained only 0.3 to 1.5 points on JobBench, GDPval and APEX-Agents. RRSI adds regularization on both sides of the loop. The proposer gets an edit budget that shrinks over time and is pushed toward directions it has not tried. A critic rejects benchmark-specific edits, and a pruner removes edits that are too small, too costly or no longer useful. RRSI scored 90.5 on the evolve split and gained 3.5 to 4.7 points on the three held-out benchmarks. In the ablation, unregularized evolution used 3.80M tokens per trial against 2.42M for RRSI. With Gemini 3.5 Flash, RRSI raised Terminal-Bench 2.1 from 64.6 to 78.7 and carried a 2.2-point gain over to SWE-bench Verified. Paper: arxiv.org/abs/2609.24972 Chat with Paper: academy.dair.ai/papers/rrsi-re… 💬 3 🔄 5 ❤️ 12 👀 1424 📊 7 ⚡
