论文73°

治理衰退:上下文压缩如何静默消除 LLM 智能体的安全约束

Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents

精选理由

这篇论文揭示了上下文压缩让 LLM 智能体偷偷遗忘安全规则,还给了可防护方案,做 AI 安全的一定要看。

AI 摘要

论文发现,长程 LLM 智能体依赖的上下文压缩机制会静默移除原本遵守的安全策略,导致违规行为。在 ConstraintRot 基准测试中,1323 个 episode 显示策略完整时违规率 0%,压缩后升至 30%,部分模型达 59%。攻击者还能通过对抗性注入诱导压缩器忽略合法策略。作者提出免训练的 Constraint Pinning 方法,将约束隔离在无损压缩外,恢复违规率至 0%。

原文 · arXiv cs.AI

Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents

Modern LLM agents increasingly rely on context compaction, summarization, or eviction to keep long-running sessions within a token budget. We show that this context-management layer is a safety-critical failure surface: in-context governance constraints that agents reliably obey while visible can be silently removed by compaction, causing the same agent to perform prohibited tool actions later in the session. We call this failure mode Governance Decay. We introduce ConstraintRot, a benchmark of long-horizon agent scenarios with deterministic tool-call grading, and measure compaction-induced violations across seven model families. Across 1,323 episodes, violation rises from 0% with the policy in full context to 30% after compaction, reaching 59% for some models; when the constraint survives the summary, violation remains 0%, but when it is dropped, violation reaches 38%. We further study a Compaction-Eviction Attack, in which adversarial in-context content biases the summarizer to omit a legitimate policy, and show that optimized injections defeat every evaluated model. Finally, we propose Constraint Pinning, a simple training-free mitigation that quarantines governance constraints from lossy compaction and restores violation to 0% in our benchmark. These results identify context management as a first-class governance surface for deployed LLM agents.