ActiveSaddler boosts agent harness optimizers via adaptive scenarios
The key insight is that optimizer quality alone is not the bottleneck in agent harness training — scenario scheduling is, and treating failure patterns as a bandit problem recovers gains the fixed-order baseline leaves on the table. Builders training coding or task agents can apply the same scheduling idea without changing their optimizer. Compared with GAIA2's 4.4-point gain, the 7.5-point jump on Terminal-Bench 2.0 suggests the method helps most where failure modes are more diverse.
Microsoft and collaborators published a paper on optimizing agent harnesses, introducing a method called ActiveSaddler.
Existing harness optimizers keep training scenarios fixed, so feedback comes from tasks that stop being informative as the harness improves.
ActiveSaddler groups recurring failures into failure patterns and treats each pattern as an arm in a non-stationary bandit, tracking how much the harness still learns from each pattern.
It splits budget between revisiting known weaknesses and finding new ones while adapting scenarios.With the same optimizer, test Pass@1 improves by 4.
4 points on GAIA2 and 7.5 points on Terminal-Bench 2.
0 compared with a fixed scenario order.
Great paper from Microsoft and colleagues on optimizing agent harnesses.
Current harness optimizers change how the harness is updated but keep the training scenarios fixed, so feedback keeps coming from tasks that stop being informative as the harness improves.
This work adapts the scenarios as well.
ActiveSaddler groups recurring failures into failure patterns and treats each pattern as an arm in a non-stationary bandit.
It tracks how much the harness is still learning from each pattern and splits the budget between revisiting known weaknesses and finding new ones.
With the same optimizer, test Pass@1 improves by 4.4 points on GAIA2 and 7.5 points on Terminal-Bench 2.0 compared with a fixed scenario order.
Paper: https://t.co/7HgcTkAReK
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include ActiveSaddler, Microsoft, GAIA2, Terminal-Bench 2.0.
Source attribution: curated from DAIR.AI (@dair_ai).