做 LLM 智能体安全评估的团队终于有了自动化工具——VESTA 能生成上千个真实任务场景，直接测出模型执行中的安全漏洞。建议关注智能体安全的开发者点开看看，结果可能会让你重新审视现有模型的风险。
VESTA 是一个全自动化的 LLM 智能体安全评估框架，能基于五个风险维度生成 1072 个可执行的评估场景。现有评估依赖人工编写场景或静态提示，难以捕捉智能体在任务执行中的多样化风险。VESTA 通过自动化流程对 12 个 LLM 智能体进行测试，发现平均安全风险率高达 47.1%，部分模型超过 70%。该框架强调了可执行、过程级评估对于理解和提升智能体安全性的重要性。
VESTA: A Fully Automated Scenario Generation and Safety Evaluation Framework for LLM Agents
Large language models (LLMs) are increasingly evolving from simple text-based interaction systems into LLM agents that can maintain memory, use tools, access external environments, and execute tasks. As their capabilities and autonomy expand, the safety risks they face also become more diverse. Existing evaluations often rely on manually written scenarios, static prompts, or final-output judgments, making it difficult to capture the diverse risks that agents may face during task execution. We introduce VESTA, a fully automated scenario generation and safety evaluation framework for LLM agents. Based on five risk dimensions, VESTA instantiaes abstract and diverse safety risks in real-world task execution into 1,072 measurable evaluation scenarios. Using the automated evaluation pipeline, 12 LLM agents are evaluated under two authority contexts. The results show that current agents still face substantial behavioral safety risks during task execution, with an average ASR of 47.1% and several models exceeding 70%. These findings demonstrate the importance of executable, process-level evaluation for understanding and improving LLM agent safety.