这篇论文告诉你那些牛逼的编程智能体排行榜可能不靠谱——参考补丁在不同机器上结果都不一样，排名还受评分规则摆布。做相关研究的人必读。
该研究审查了GSO、SWE-Perf和SWE-fficiency三个代码优化基准，发现740个任务中参考补丁在不同Google Cloud机器上重复满足有效性规则的比例分别仅为39/102、11/140和411/498。SWE-Perf尤其脆弱，许多参考补丁产生接近零的运行时间变化。在8个公开提交中，9/28对提交的官方排名不一致，SWE-fficiency的评分规则将最差10个任务的权重提高到58.5%-82.8%。在450个可重复任务中，85.3%的任务至少有1个公开提交匹配或超越参考补丁，99.8%的任务优于未优化基线。该研究揭示了聚合排名掩盖的剩余性能差距。
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.