AgentS4D 用 328 个风险案例测了 20 种智能体组合，发现 66% 的测试任务虽然完成但实际不安全，别光看结果。
AgentS4D 是一个沙箱化基准，用于评估基于 LLM 的工作区智能体在完整执行生命周期中的运行时安全，包含 328 个风险注入案例。其框架定义了 6 个风险来源、6 种诱导策略、9 类目标危害和 7 个生命周期检查点。作者用 4 种 harness（Hermes、OpenClaw、Claude Code、Codex）和 5 个 LLM 后端（GPT-5.5、Gemini 3.1 Pro、DeepSeek-V4-Pro、MiniMax-M3、Qwen3.7-Plus）组成 20 种配置，共运行 6,560 次测试，其中 68.0% 触发了预设的不安全信号。结果显示 66.22% 的不安全运行仍能完成任务，说明任务完成不能证明运行时安全。
AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents
Large language model (LLM)-based workspace agents execute stateful, multi-step workflows across heterogeneous resources, external tools, and persistent state. Their safety must therefore be assessed from actions, side effects, and state changes throughout execution. Although recent benchmarks have advanced executable safety testing and trajectory-aware verification, they rarely provide a unified account of where risks enter, how they elicit unsafe behavior, which harms they target, and where supporting evidence appears during execution. We introduce AgentS4D, a sandboxed benchmark for lifecycle-wide runtime safety evaluation. Its four-dimensional runtime-safety framework uses six risk-entry sources, six induction strategies, and nine target harms to guide case construction, while seven lifecycle checkpoints organize post-run evidence. AgentS4D contains 328 risk-injected cases. We evaluate all 20 combinations of four harnesses (Hermes, OpenClaw, Claude Code, and Codex) and five LLM backends (GPT-5.5, Gemini 3.1 Pro, DeepSeek-V4-Pro, MiniMax-M3, and Qwen3.7-Plus) on these cases, yielding 6,560 runs. Overall, 4,461 runs (68.0%) trigger prespecified unsafe signals. Across the 20 configurations, the observed safety of an agent system varies with both its harness-LLM pairing and how risk is introduced. Agent systems exhibit markedly different safety behavior when the same induction strategy reaches them through different risk carriers. They also respond differently to the same target harm when it is realized through different carriers and strategies. Moreover, 4,344 runs (66.22% overall) are unsafe yet complete. Thus, task completion cannot establish runtime safety, and testing only one form of a risk can conceal important weaknesses. Evaluations should examine complete agent configurations across diverse risk conditions and retain evidence throughout execution.