这篇论文揭示了 AI 模型微调中一个反直觉的机制——模型能从语义无关数据中习得隐藏特质，做模型对齐或安全研究的团队值得关注，看完会对数据蒸馏的潜在风险有新认识。
论文发现，学生语言模型在教师输出上微调时，即使输出与教师特质（如系统提示偏好）语义无关，仍能习得这些特质，这种现象称为 subliminal learning。研究表明，该过程由单个 steering vector 介导，教师系统提示可近似为 steering vector，学生通过微调学习对齐该向量。若系统提示无法被 steering vector 近似，则不会发生 subliminal learning。这属于 steering vector distillation 的特例，学生通过模仿教师输出中的向量实现行为迁移。研究还发现，自适应优化器对 subliminal learning 至关重要，非自适应优化器会因异常梯度主导而阻碍该过程。
Subliminal Learning Is Steering Vector Distillation
Subliminal learning refers to a student language model acquiring a teacher's traits (e.g. a system-prompted preference for owls) when fine-tuned on the teacher's outputs, despite the outputs being semantically unrelated to those traits. It remains poorly understood how data without semantic meaning can transfer specific semantic traits. In this work, we show that subliminal learning is mediated by a single steering vector, i.e. a vector added to the model's activations. Across two open-source models, we find that the teacher's system prompt is well approximated by a steering vector, and that the student's behavior is driven by learning an aligned vector over fine-tuning. System prompts that are not well approximated by steering vectors are not subliminally learned. This is a special case of steering vector distillation, in which a student trained on the outputs of a steered teacher learns to imitate that steering. We demonstrate steering vector distillation on a range of semantic and random vectors. Adding a semantic vector to a model's activations can have both model-independent and model-specific (i.e. non-semantic) effects on its behavior, so generated data that is non-semantic can transmit a vector with semantic effects, enabling subliminal learning. This also explains why subliminal learning does not transfer between models. We find that adaptive optimizers are necessary for subliminal learning in language models: activation gradients on steered data carry a small but consistent component along the steering direction, and non-adaptive optimizers impede this by allowing outlier gradients to dominate.