Papers

arXiv 论文提出"社会化对齐":用演化选择研究智能体行为权衡

Alignment by Socialization: Learning Tradeoffs Under Selection

Why we picked it

这篇论文把对齐问题从"给目标"变成"环境怎么塑造行为",用演化选择的数学模型推得很清楚。做智能体安全、RLHF 或长期部署的人值得细看,尤其是社会化可逆性和硬性禁令代价那部分的结论。

arXiv 论文(编号 2611.18531)提出"通过社会化实现对齐"(Alignment by Socialization)框架,用演化选择模型研究智能体在完成任务与遵守行为规范之间的权衡。模型中多个智能体副本面对共同的不确定奖励状态,只有采取被奖励动作的个体才能繁殖,长期增长率在动作采取频率与环境奖励频率匹配时达到最大。

Body

论文对比三种策略:仅任务策略在只奖励任务完成的沙盒中表现良好,但在行为规范重要的环境中失效;固定线性策略会随着任务利害上升而越来越偏向完成任务,即使人类从不奖励这种偏移;学习型策略则收敛到环境所奖励的权衡——当奖励由人类提供时,作者称之为"社会化"。实验用小型神经网络演示训练目标和记忆如何影响结果。

关键发现包括:积累的社会化经验使智能体在后续仅任务训练中更难逆转既有倾向,但违规行为可能在主导倾向改变之前就出现,且有限记忆会削弱这种保护。在单次违规即终止部署的场景下,硬性禁令可以防止违规,但错误的禁令本身也有代价。

Original · arXiv cs.AI

Alignment by Socialization: Learning Tradeoffs Under Selection

Artificial intelligence (AI) agents are expected to complete tasks while following rules of conduct, but these objectives can conflict. We study how evolutionary selection shapes an agent's choices between them, and whether experience can make those choices reflect human judgments and withstand subsequent pressures. In our model, copies of an agent face a common uncertain reward state, and only those taking the rewarded action reproduce. We show that long-run growth is maximized when each action is taken as often as the environment rewards it. We compare three policies. A task-only policy thrives in a sandbox rewarding completion alone but fails when conduct matters. A fixed linear policy increasingly favors task completion as its stakes rise, even where humans never reward it. A learned policy instead converges to the tradeoffs rewarded by its environment; when humans supply the rewards, we call this socialization. Accumulated experience makes socialization harder to reverse under subsequent task-only training, although violations can arise before the prevailing disposition changes, and finite memory weakens this protection. When a single violation can end a deployment, a hard prohibition can prevent it, but a mistaken prohibition can also be costly. Experiments with small neural networks illustrate how training targets and memory shape these outcomes. Our results suggest that alignment depends not only on the objectives assigned to agents, but also on the environments that shape and sustain their behavior.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Alignment, AI safety, arXiv, agents.

Source attribution: curated from arXiv cs.AI (@Thomas Brennan, Andrew W. Lo, Chaoyi Zhao).