论文:多智能体协作会稀释安全对齐,指定安全主管可修复
Aligning AI Teams
做智能体团队开发的人该看看这篇。它用 12 个任务证明让智能体互评、开群聊这些常见做法救不回安全对齐,真正有效的是设一个安全主管智能体加一份安全规格文档。作者还开源了 magelab 框架,可以直接拿来做实验。
arXiv 论文(编号 2611.17820)研究了单个 LLM 智能体的对齐训练在多智能体团队中失效的机制。
实验覆盖脓毒症分诊、新闻推荐两个软件工程任务和 10 个咨询提案生成任务,测试了包括 Anthropic Mythos Preview 在内的多个模型。结果发现单智能体与团队的安全差距不随模型能力提升而稳定缩小:Mythos Preview 在一个任务上差距最大,在另一个任务上反而最小。
让智能体互相批评或在群聊中协调等常见干预手段,均无法稳定恢复单智能体水平的对齐行为。作者定位到核心原因是责任扩散:团队中的智能体更少主动检查系统级安全故障,出问题时更容易忽视、合理化或推卸责任。
有效的修复方法是设立专门的安全主管智能体负责系统级安全,并由协调者智能体发布安全规格文档。论文同时开源了多智能体编排实验框架 magelab。
Aligning AI Teams
Previous work has shown that alignment training on individual LLM agents can fail to transfer to multi-agent settings. We investigate the underlying mechanisms for this failure, and explore interventions to preserve alignment transfer to multi-agent LLM teams. We focus on two software-engineering tasks (sepsis triage, a news recommender) and ten consulting-proposal generation tasks. We present four findings: (1) The single-vs-team safety gap does not show a clear trend with more capable models. We find that Anthropic's Mythos Preview shows the largest gap on one of our tasks but the smallest on another; (2) Many natural interventions, such as having agents critique each other's work or coordinate in a group chat, do not reliably bring agent teams back to single-agent alignment behavior; (3) diffusion of responsibility substantially accounts for the safety gap: agents in teams are less likely to proactively check for system-level safety failures and more likely to ignore, rationalize, or deflect responsibility for issues when they arise; (4) having a designated safety lead agent ensure system-level safety, and having a coordinator agent publish a safety specifications document are effective at restoring single-agent alignment. We also release magelab, a multi-agent LLM orchestration and experimentation framework. Our findings can help practitioners design agent teams that successfully preserve the behaviors expected of aligned individual agents.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include multi-agent systems, Anthropic, AI safety, magelab.
Primary source verification: published directly by arXiv: Anthropic with first-party authority.