研究：SWE-bench 上通过测试的补丁近半数存在运行性能差异
Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution
四个前沿模型解 SWE-bench，通过测试的补丁一半有性能差距，模型性能排名还和解决率反着来，做工程的人可以看看。
arXiv 论文 2610.13563 用 SWE-agent 驱动 Claude Opus 5、GPT 5.6、DeepSeek V4 Pro 和 Qwen 3.8 Max 处理 SWE-bench 与 SWE-bench-Live verified 中筛出的 262 个实例，每个实例至少有两个正确解法。研究复用基准自带的 FAIL_TO_PASS 测试来测补丁的指令数和墙钟时间，并用 Welch ANOVA 替代先前工作的 RSD 作统计判据。结果显示约一半实例的补丁在运行性能上有显著差异，且性能排名与各模型的问题解决率几乎相反。案例分析还发现部分补丁存在正确性测试未覆盖的架构级性能回退。
Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution
Repository-level issue-resolution benchmarks such as SWE-bench evaluate submitted patches primarily based on whether they pass the correctness tests, leaving the runtime performance of different test-passing patches largely unexplored. Evaluating patch performance is essential but challenging since designing performance workloads for each benchmark instance is costly. However, these benchmarks already provide a natural candidate: the built-in \emph{FAIL\_TO\_PASS} (F2P) tests, which exercise the behavior affected by the issue fix. We therefore investigate whether the provided F2P tests can be repurposed to evaluate patch performance. Overall, we choose four frontier models, Claude Opus~5, GPT~5.6, DeepSeek V4 Pro, and Qwen~3.8 Max, paired with SWE-agent to resolve the pre-filtered measurable corpus from verified splits of SWE-bench and SWE-bench-Live, yielding 262 cases with at least two different correct solutions, including the developer ground-truth patch. We then measure each solution's runtime performance in instruction count and wall-clock time on the native F2P test, and use Welch ANOVA instead of the RSD proposed in prior work as a more robust statistical discrimination criterion. We find that this methodology exhibits notable discrimination capability in runtime performance and identifies performance differences among patches in almost half of the instances. Our empirical study shows that the runtime performance of generated patches is independent of the issue resolution rate of LLMs, with nearly reversed model rankings. The case study also shows that the performance analysis reveals architectural regression in some patches that the correctness evaluation overlooked. These findings suggest that resolution rate does not adequately characterize LLMs' capability for repository-level issue resolution, motivating future evaluations to consider patch performance alongside it.
