PapersSelected

ActiveSaddler: Microsoft Paper on Automated Agent Harness Optimisation

Why we picked it

Choosing training tasks by remaining failures, rather than a fixed list, cuts optimiser spend by roughly 78% while lifting pass rates — task selection is a bigger lever than prompt or tool patching. Builders who auto-tune agent harnesses can redirect their run budget toward unresolved failure patterns. GAIA2 and Terminal-Bench 2.0 numbers give a direct benchmark for comparing harness optimisation setups.

Microsoft published a paper on automated harness optimisation for AI agents, introducing ActiveSaddler.

Body

Unlike tuners that use a fixed task list, ActiveSaddler tracks failure patterns and selects the task most worth fixing, or probes unseen tasks to discover new failure modes.

On the same optimiser, it raised test pass rates by 4.4 points on GAIA2 and 7.

5 points on Terminal-Bench 2.0.

Reaching 58.5% GAIA2 dev accuracy cost US$298, versus US$1,360 with a fixed task order.

Original · rohanpaul_ai

New Microsoft paper on Automated harness optimization for agents.

Most harness auto-tuners focus on how to patch prompts and tools, but which tasks produce the feedback also changes how good the final harness gets.

But you will get stronger AI agents when you pick training tasks based on which failures are still unfixed, so stop feeding them a fixed task list.

ActiveSaddler tracks failure patterns and works on the one most worth fixing, or tries unseen tasks to find new ones.

On the same optimizer, ActiveSaddler raised test pass rates by 4.4 points on GAIA2 and 7.5 points on Terminal-Bench 2.0. Reaching 58.5% GAIA2 dev accuracy cost $298, versus $1,360 with a fixed order.

If you auto-tune an agent, aim your run budget at the failures that are still open.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Microsoft, ActiveSaddler, GAIA2, Terminal-Bench.

Source attribution: curated from rohanpaul_ai (@rohanpaul_ai).