ParetoSaddler optimizes agent harnesses for quality and cost
The design lesson is that multi-objective optimization in agent pipelines can backfire when applied too early — deferring the cost criterion until final selection preserves valuable intermediate candidates. Builders running agent harness or prompt searches should add repeated runs and a cost axis to their selection step. The arXiv paper (2611.02079) and dair.ai's Chat with Paper make it easy to replicate the setup.
Microsoft's ParetoSaddler applies a cost preference only at the end of harness search, rather than at every step, so candidates that lead to better harnesses are not discarded. An optimizer edits a parent harness, both versions run several times, and a Bayesian bootstrap decides whether the candidate shows credible progress in quality or cost. On GAIA2, quality rises from 56.22% to 62.89% while cost per task falls from $0.78 to $0.45. On ClawMark, quality improves from 70.86% to 81.35% with cost dropping from $1.17 to $1.01. Pareto-selected harnesses beat quality-only optimization on both metrics.
Super interesting finding on optimizing agent harnesses. When you optimize an agent harness for both quality and cost, apply your cost preference only at the end of the search, since applying it at every step throws away harnesses that lead to better ones. That is the key insight behind Microsoft's ParetoSaddler. At each step, an optimizer edits a parent harness. Both versions run several times, and a Bayesian bootstrap over those runs decides whether the candidate shows credible progress in quality or in cost. Promoted candidates stay available as parents, and the final deployment choice is made separately. On GAIA2, quality goes from 56.22% to 62.89% while cost per task drops from $0.78 to $0.45. On ClawMark, quality goes from 70.86% to 81.35%, and cost drops from $1.17 to $1.01. The Pareto-selected harnesses also beat quality-only optimization on both quality and cost. If your harness search only tracks accuracy, add repeated runs and a cost axis before deciding which candidates to keep. arxiv.org/abs/2611.02079 Chat with Paper: academy.dair.ai/papers/paretos… 💬 24 🔄 5 ❤️ 30 👀 3804 📊 28 ⚡