研究指出编译率不能可靠评估 LLM 代码漏洞修复效果
Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
一篇打脸编译率的论文：64% 编译失败根本不怪模型，改个编译选项编译率就差 2 倍多，做代码修复评估的都该看看。
一项实证研究针对 203 个来自 Big-Vul 的 C/C++ 漏洞函数，测试了三个参数量 350M 到 6.7B 的开源代码 LLM 和三种提示策略。结果显示编译率作为评估指标不可靠：约 64% 的编译失败并非模型原因，且同一补丁仅改变一个编译器标准选项就会使编译率波动 1.8 到 2.7 倍。把编译率作为优化目标还会奖励“非修复”——编译反馈循环让编译率上升，但与人类修复的相似度下降，人工检查发现了删除式和占位符式的假补丁。研究者还发现整函数 CodeBLEU 也失效，未修改的漏洞输入副本得分反而高于所有模型，并提出 diff_F1 作为只对编辑区域打分的低成本筛查指标。
Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.