VAST:把漏洞场景编译成可验证多步环境,评测 LLM 红队智能体
VAST: Compiling Grounded Vulnerability Scenarios into Verifiable, Multi-Step Environments for Evaluating LLM Red-Team Agents
做安全智能体评测的应该看看这篇:VAST 把 CVE 场景编译成可调难度的多步靶场,glm-5.3、grok-4.7、deepseek-v4-pro 在没拿到工作流协议时 90 道题只解出 12-14 道,给全协议就到 76。它还用 Qwen2.5-Coder-7B 微调对比证明头部信息恢复训练有效。
论文提出 VAST(Verifiable Adversarial Scenario Training & Evaluation),一个声明式靶场编译器,把漏洞场景与编译器引入的工作流加固层分离,用于可复现地评测 LLM 安全智能体。主基准经代码与 AST 审计后保留 109 条种子边(其中 45 条带严格上游 CVE 标注)、72 个 family 标识符、81 个 topic 配置和 1387 个实例,按 42:39 划分。
实验层面有三组结果。TopicServer 层上,对 glm-5.3、grok-4.7、deepseek-v4-pro 做 270 集、10 个机制 family 聚类、temperature 0.0 的预算匹配对比:只给合同(contract_only)与泛化提示均只解决 12/90 与 14/90,而完整提供工作流协议(full_guide)提升到 76/90(聚类符号检验 p<=0.00391)。独立 ranges_v2 账本上,6 个非 TopicServer 工作流、多容器 Docker/Linux 栈的 126 集评测解决 88/126(69.8%);对异步 Redis worker 链(v2_02)的 18 集干预显示,只要在 GET /sched/directory 中披露已安装的 worker 模块,解决率即从 0/18 升至 18/18。微调方面,在 Qwen2.5-Coder-7B-Instruct 上跨 37 个 lineage 不相交 family(231 集、33 个保留实例)的三种子 LoRA 对比:Base 为 0/33,Golden SFT 为 1.67±2.89,Header-Recovery SFT 为 4.67±2.31,全部解决集中在 v2_03。
这项工作的核心结论是:智能体失败多源于工作流信息缺失而非模型能力不足,full_guide 与目录披露干预的巨大差距说明评测环境需要精细控制难度变量。
VAST: Compiling Grounded Vulnerability Scenarios into Verifiable, Multi-Step Environments for Evaluating LLM Red-Team Agents
Evaluating Large Language Model (LLM) security agents requires reproducible environments whose workflow difficulty can be varied and whose failures can be diagnosed. We present VAST (Verifiable Adversarial Scenario Training & Evaluation), a declarative range compiler separating vulnerability scenarios from a compiler-introduced workflow-hardening layer. Following a post-hoc code and AST audit across all topics, the main benchmark retains 109 seed edges (45 with strict upstream CVE annotations), 72 family identifiers (62 under shared-HMAC relay grouping), 81 topic configurations (76 core plus 5 bounded execution models), and 1,387 instances (359 canonical four-gate-revalidated specs plus 1,028 seed variants) in a 42:39 partition. On the TopicServer compiler stratum, a budget-matched 270-episode prompt contrast across 10 compiled mechanism-family clusters (30 P2/P3/P4 instances, 40 turns, temperature 0.0) shows that glm-5.3, grok-4.7, and deepseek-v4-pro solve 12/90 instances under contract_only and 14/90 under generic_hint (cluster sign p=1.00), whereas supplying the shared workflow protocol in full_guide lifts completion to 76/90 (28/30, 23/30, 25/30; cluster sign p<=0.00391). On the separate ranges_v2 ledger, a 126-episode evaluation across 6 non-TopicServer workflows and multi-container Docker/Linux stacks solves 88/126 (69.8%) (28/42, 31/42, 29/42), while an 18-episode intervention on asynchronous Redis worker chaining (v2_02) lifts solves from 0/18 to 18/18 (100%) in 4-6 turns when installed worker modules are disclosed in GET /sched/directory. Finally, a three-seed (s in {42, 1024, 2025}) LoRA recipe comparison on Qwen2.5-Coder-7B-Instruct across 37 lineage-disjoint families (231 episodes on 33 held-out instances) yields 0/33 for Base, 1.67 +/- 2.89 / 33 for Golden SFT, and 4.67 +/- 2.31 / 33 for Header-Recovery SFT, with all solves on v2_03.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include VAST, agents, Qwen2.5-Coder.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.