四个安全智能体实测真实漏洞:CodeRabbit Security 以 86.7% 准确率居首
CodeRabbit 拿真实 CVE 隐藏答案考了四个安全智能体,Devin 和 Claude 的分数差距挺意外,写测试方法那部分很实在。
CodeRabbit 从 GitHub Advisory Database 提取真实漏洞测试了四个安全智能体,测试时隐藏了安全通告、CVE 编号和补丁。结果 CodeRabbit Security 以 86.7% 准确率排第一,Devin 73.3%,Codex Security Plugin 62%,Claude Security 40%。评分规则严格:结论必须命中真实漏洞利用路径和正确位置才算分,CWE 类型对但路径错记零分。
We gave four security agents real vulnerabilities from the GitHub Advisory Database, with the advisory, CVE and patch withheld.
> CodeRabbit Security: 86.7% > Devin: 73.3% > Codex Security Plugin: 62% > Claude Security: 40%
A finding only scored if it matched the real exploit path at the right location.
Right CWE, wrong path: zero.
Real our full write up below!