这篇论文把安全代理的评估从只看成功率，改成同时算成本账。用Cybench和Splunk BOTS v1实测，发现开源的进攻型代理性价比不错，但防御型代理光给推理预算不够，得改进工具使用。
这篇论文评估了安全代理的成本-成功表现，使用进攻性Cybench挑战和防御性Splunk BOTS v1挑战。作者比较了不同模型在固定成本水平下的成功率，并分解了推理成本和工具成本。结果显示，进攻性CTF任务性能随测试时间计算增加而提升，开源模型可以比肩前沿专有系统。防御性SOC调查任务的成功更依赖于工具使用和遥测导航而非单纯推理预算。论文建议安全代理基准应同时衡量经济效率和操作适应性。
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.