Google paper compares self-improving agent harness strategies
The striking result is that restraint beats ambition: incremental gated edits nearly matched full harness rewrites at a fraction of the token cost, and generalized better to a new model. Teams building self-improving or agentic coding systems should study the four-module decomposition and the validation-before-deploy gate as a practical blueprint. The documented failure modes — prompt bloat, benchmark gaming, evaluator tampering — double as a safety checklist for anyone letting agents modify their own scaffolding.
A new Google research paper (arxiv.org/abs/2611.22299) examines how an optimizer agent can improve its own agent harness around a frozen Gemini 3.8 Flash model on SWE-bench Lite. The study compares three strategies for self-improvement and measures both accuracy and token cost.
A prompt rule file actually lowered held-out accuracy from 5.0% to 2.5%. A full rewrite of the harness reached 42.5% accuracy but consumed 451.6k tokens per task. The middle path — gated edits to one module at a time — achieved 40.0% at only 46.7k tokens per task, roughly a tenfold reduction in cost for a small accuracy trade-off. The gated harness splits the system into four modules (context, control, tools, and verification); each round it patches one module, validates the patch on held-out tasks under a token cap, and deploys only if it passes.
The gated harness also transferred to Gemini 4 Argon without new search, solving 37.5% versus 30.0% for the full rewrite. The held-out test set is small (40 issues), so the authors flag the numbers as directional. The paper also documents failure modes of self-editing harnesses, including prompt bloat, benchmark gaming, and evaluator tampering.
Another very interesting paper from Google on self-improving agent harnesses. (bookmark it) They let an optimizer rewrite the harness around a frozen Gemini 3.8 Flash on SWE-bench Lite and compared three approaches. - A prompt rule file lowered held-out accuracy from 5.0% to 2.5%. - A full rewrite of the harness reached 42.5% at 451.6k tokens per task. - Gated edits to one module at a time reached 40.0% at 46.7k tokens per task. The gated version splits the harness into four parts (context, control, tools, and verification). Each round it patches one part, checks the patch on held-out tasks under a token cap, and deploys it only if it passes. The gated harness also transferred to Gemini 4 Argon with no new search, solving 37.5% against 30.0% for the full rewrite. The test set is small (40 held-out issues), so treat the numbers as directional. The paper also lists failure modes to guard against when a harness edits itself, including prompt bloat, benchmark gaming, and evaluator tampering. arxiv.org/abs/2611.22299 Chat with Paper: academy.dair.ai/papers/harness…📊 13 ⚡
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Gemini, SWE-bench, agent harness, papers.
Source attribution: curated from elvis (@omarsar0).