SWE-bench audit shows coding agents can't be ranked
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
This matters to developers because it shows that the current leaderboard doesn't provide a reliable way to compare coding agents, which could lead to misinformed decisions about which tool to use.
A new audit of the SWE-bench coding agent leaderboard found that small differences in scores don't mean one system is better than another.
The leading two models on the Verified split both solved 396 out of 500 test cases.The audit found that model-scaffold pairs have a much wider performance spread than the top models, with a 29.
8 percentage point difference compared to 8.8 points for the top thirty.
The study recommends reporting comparison-specific resolution and model-scaffold provenance instead of interpreting small score gaps as definitive rankings.
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Frontier solution sets have median nesting 0.935 against a score-implied baseline of 0.774, indicating strongly shared successes. Scores also depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction; non-rejection does not establish equivalence. We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution. The results motivate reporting comparison-set-specific resolution and model-scaffold provenance instead of interpreting small aggregate gaps as established rank differences.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include SWE-bench, coding agents, leaderboard, benchmark.
Source attribution: curated from arXiv cs.AI (@Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo).
