这篇研究不只看模型能修复多少问题，还比较了Claude和DeepSeek不同代际的代码质量、CPU时间和内存使用，发现新模型虽修得多但代码质量没明显提升。
这篇论文通过两个案例研究分析了Claude和DeepSeek的四代模型在SWE-bench Lite上的非功能性质量差异。静态分析显示大多数CodeQL配对差异为零，且无CodeScene比较在Holm校正后显著。CPU时间差异较小且不一致，峰值内存略高但绝对差异很小。整体上，较晚模型解决更多实例，但在共同解决的任务中非功能性指标无一致改进。
Beyond Resolved Rate: A Non-Functional Quality Study
Repository-level coding benchmarks typically measure progress in model capability by comparing the resolved rates of later and earlier models. However, this focus overlooks whether the non-functional quality of their generated patches has also changed across model generations. This study investigates whether later models produce functionally correct patches with better non-functional characteristics than earlier models on comparable repository-level repair tasks. We conducted two case studies involving four Claude and DeepSeek models on SWE-bench Lite. Using the same SWE-agent functional repair setting, we evaluated the generated patches with CodeQL, CodeScene, CPU time, and peak memory. Our primary analysis compared the models on commonly resolved instances. The static analysis results showed that most CodeQL paired differences were zero and that no CodeQL or CodeScene comparison remained significant after Holm correction. CPU time differences were small and inconsistent across model families, while peak memory usage was slightly higher for the later models under the benchmark test workload, with small absolute differences. Differences in individual CodeQL rules and CodeScene categories varied across model families and did not survive multiple-comparison correction. Overall, later models resolved more instances but showed no consistent improvement in the measured non-functional indicators on tasks solved by both models. Through this study, we hope to encourage a more comprehensive evaluation of models' practical software engineering capabilities.