Vulnerability Localization Benchmark for AI Agents
Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
This research provides a new standard for evaluating how well AI systems can find and fix security vulnerabilities in large codebases, which is important for improving the security of software development.
A new benchmark called VLoc Bench measures how well AI agents can find the code related to a specific software vulnerability.
It uses 500 real-world vulnerabilities from 290 repositories.The benchmark tests 27 language models and four static-analysis tools.
The best system only achieved a 0.229 File F1 score, meaning it was not very good at finding the correct files.
Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code. We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness. We introduce the Vulnerability Localization Benchmark (VLoc Bench), comprising 500 real world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories. Each task pairs repository snapshots immediately before and after a security fix. On the vulnerable snapshot, an agent receives only the CWE description and read-only terminal access and must return the affected files; on the patched snapshot, it must determine that the recorded vulnerability is no longer present. We evaluate 27 language models and four static-analysis tools under a common agent interface. Repository-scale vulnerability localization remains difficult: the strongest system achieves 0.229 File F1, and 38.4% of tasks receive no correct localization from any evaluated model. We further find that stronger localization does not imply reliable behavior after remediation: systems that identify vulnerable files effectively can still report unsupported locations on patched repositories. These results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include VLoc Bench, vulnerability localization, CWE, software security.
Source attribution: curated from arXiv cs.AI (@Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, Arthur Goldblatt, Zhuoran Yang, Amin Karbasi).
