这篇把对齐提前到了预训练第一步，3B 模型跑 500B token，越狱更难破，值得搞 AI 安全的人看一眼。
论文提出合成人格预训练（SPP），在预训练阶段就从第一个 token 开始植入目标助手人格，而非等预训练后叠加价值观。方法先用价值规范给预训练文档标注第一人称反思，再在标准文档和反思上以交叉熵损失训练，最后用对话数据做人格绑定。在 3B 参数、500B token 的预训练中，SPP 提高了宪法遵循和越狱鲁棒性，降低了分布外道德困境中的错位率，同时保持能力。实验显示，仅在预训练末期引入 SPP 效果明显更弱，且优势随预训练预算增加而增大。
Synthetic Persona Pretraining: Alignment from Token Zero
As language-model-based AI is increasingly deployed in autonomous settings, aligning its goals and values with those of humans becomes critical. Today, alignment, and the assistant identity itself, are typically introduced only after pretraining, once behavioral priors are already established. This can make values a thin overlay, rather than deeply rooted, and facilitate subsequent misalignment. Pursuing a different paradigm, we introduce Synthetic Persona Pretraining (SPP), which installs the desired assistant persona from token zero in pretraining. First, we annotate pretraining documents with value-aligned first-person reflections derived from a normative value constitution. Second, we pretrain via the standard cross-entropy loss on standard pretraining documents as well as their reflections, which installs the desired persona among a multitude of other personas. Finally, we post-train on user-assistant dialogue data, which binds this desired persona to the assistant identity, a process we call persona binding. By pretraining models up to 3B parameters on 500B tokens, we show that SPP improves constitution following and jailbreak robustness, and reduces the misalignment rate in out-of-distribution moral dilemmas, while preserving capabilities. Early intervention matters: compared with alignment from token zero, introducing SPP only at the end of pretraining yields weaker constitution adherence, does not shift value priorities, and leads to less aligned choices in dilemmas. This advantage depends on persona binding and, importantly, increases with pretraining budget. Overall, our results show that shaping values early is critical for alignment and establish pretraining-time persona interventions as an effective approach to do so.