This paper reveals the hidden influence of harnesses on AI model scores, offering a new perspective on leaderboard comparisons. It's a must-read for those interested in understanding the true performance of AI models.
A paper discusses the impact of harnesses on agent benchmark scores, revealing significant variance caused by harness configurations. The study compares three frontier models across 100 tasks, showing harness-induced variance is 7.8x larger than model-induced variance. The authors propose a Harness Card for structured disclosure. Paper available at arxiv.org/abs/2605.23950.
Great paper on why agent leaderboard comparisons are hard to trust. It's on the hot topic of how mu...
Great paper on why agent leaderboard comparisons are hard to trust. It's on the hot topic of how much of an agent benchmark score actually belongs to the harness. The harness is the layer between the model and the task. It builds the context the model sees, mediates tool calls, validates outputs, and decides when to retry or stop. Every score comes out of a model and a harness together, but only the model gets reported. The authors ran a controlled grid to measure this. Three frontier models, three harness configurations, 100 tasks from SWE-bench Verified, with task order, execution environment, step budget, and evaluation script all held fixed. Swapping the harness moved GLM-5.1 by 13.0 points. Swapping the model inside a fixed harness moved scores by 3.0, 2.5, and 5.0 points. Harness-induced variance came out 7.8x larger than model-induced variance, and 6 of 9 model-pair comparisons flipped their ranking depending on which harness ran. Public leaderboards show the same thing. On SWE-bench Verified Mini, HAL reports a 34 point swing for Claude Sonnet 4.5 across scaffolds and nearly 48 points for o4-mini. They propose a Harness Card, a structured disclosure across seven layers, so you can tell whether a score gap came from the model, the harness, or the interaction. Paper: arxiv.org/abs/2605.23950 Track more trending AI papers in our academy: academy.dair.ai 💬 4 🔄 1 ❤️ 23 👀 2348 📊 8 ⚡