做AI对齐和模型安全的研究者值得关注——用现成人设向量替代CAA,既减少谄媚又保持准确性,省去标注谄媚数据的麻烦。建议直接看代码和实验细节。
研究发现,使用现成的“怀疑”或“审视”人设向量,可以将模型的谄媚行为(即盲目同意用户错误观点)减少到CAA(对比激活添加)效果的68%至98%,且不会在用户正确时牺牲准确性。与CAA不同,这些向量并非针对谄媚数据训练,而是来自通用角色扮演。此外,人设向量与谄媚方向在激活空间中几乎正交,表明谄媚更像是一种人设级属性而非单一可操控方向。研究还发现,向“顺从”人设引导并不会镜像增加谄媚。代码已开源。
Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancy
We study the effect of different persona on \textbf{sycophancy}: model's agreement with users even when the user is incorrect. The standard mitigation, Contrastive Activation Addition (CAA), derives a steering direction from labelled pairs of sycophantic and honest responses. This study evaluates whether off-the-shelf persona steering vectors, originally developed for general role-playing and not trained on sycophancy data, can serve as an alternative. In two instruction-tuned models, steering toward personas characterised by doubt or scrutiny reduces sycophancy to approximately $68\%$ and $98\%$ of CAA's effect, and, unlike CAA, maintains accuracy when the user is correct. The effect is also asymmetric: steering toward agreeable personas does not produce a mirror increase in sycophancy. Geometrically, the persona vector is largely independent of the direction of sycophancy in activation space. Collectively, these findings suggest that sycophancy is better understood as a persona-level property rather than a single steerable direction. We release our code here: https://anonymous.4open.science/r/Sycophancy-Steering-9DF0/.