SecPRBench：用真实漏洞提交评测智能体代码审计能力
Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?
想知道 Claude 或 GPT 审代码到底能不能抓到真漏洞？这篇基准测了八款模型，265 个来自 Firefox、OpenSSL 的真实漏洞提交，结果差异挺大。
论文提出 SecPRBench，首个基于真实 bug 引入提交的智能体安全代码审计基准，通过自动化流水线采集、构建并验证漏洞提交，目前已收集 265 个任务，覆盖 Firefox、V8、OpenSSL 等九个关键系统。评测八款前沿模型，召回率在 32.5% 到 78.1% 之间，Grok 4.6 和 GPT-6 sol 排名靠前。研究还发现 68% 的漏报案例中智能体定位到了含漏洞的文件，但给出的漏洞机制判断错误。安全专用配置表现分化：OpenAI 的 Codex Security 因更多弃答使召回率下降 8.3 个百分点，Anthropic 的 Defending Code 配置则提升 7.7 个百分点，但成本更高。
Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?
LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.