Soteria:形式化验证加运行时强制,让 LLM 智能体安全与效率兼得
Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents
做智能体开发的人可以看看这篇:它把安全检查提前到计划阶段,用形式化验证保证每一步工具调用合规。被拦截时不硬卡死,而是靠已验证计划给出恢复路径。基准上成功率是最佳基线的 1.7 倍且零违规,比常见的运行时过滤方案更系统。
论文提出 Soteria 框架,针对 LLM 智能体多步工具调用中“完成任务”与“遵守策略约束”难以兼顾的问题给出新解法。在执行任何工具调用之前,智能体先生成结构化计划,并针对形式化规约做验证,确保计划本身合规。
执行阶段,已验证的计划承担双重角色:约束智能体轨迹符合规约,同时在某个不安全动作被拦截后提供恢复指引,避免智能体陷入停滞。作者指出,现有方法(如只做拦截或只做事后过滤)往往牺牲一方目标,而 Soteria 通过分层规划与运行时强制将两者统一起来。
在覆盖多类工具使用任务的多个基准上,Soteria 的平均任务成功率达到最佳基线的 1.7 倍,且策略规约违规数为零。对构建工具调用智能体的开发者来说,这套“先验证计划、再执行并强制”的思路可直接借鉴到 MCP/工具编排与智能体框架的安全设计中。
Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents
LLM agents that execute multi-step tool calls must satisfy two objectives simultaneously: completing the user's task (utility) and conforming to domain policies and correctness constraints (safety). These objectives are in tension -- blocking an unsafe action prevents a violation but can leave the agent stranded with no principled way to recover. Existing approaches sacrifice one objective for the other. We introduce Soteria, a framework that reconciles safety and utility through verified hierarchical planning and runtime enforcement of specifications. Before execution, the agent generates a structured plan that is formally verified against the specifications prior to any tool invocation. During execution, the verified plan serves dual roles: it ensures that the agent takes specification-conformant trajectories and provides guidance when unsafe actions are blocked. Across multiple benchmarks covering a diverse range of tool-use tasks, Soteria achieves 1.7 times the average task success rate of the best baselines, with no violations of policy specifications, demonstrating that safety and utility need not be traded off.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, agents, formal verification, MCP and tools.
Source attribution: curated from arXiv cs.AI (@Mike He, Ankush Desai, Sharad Malik, Aarti Gupta).