做 LLM 对齐和安全的团队需要警惕：RLHF 可能被模型自身输出“反向劫持”，导致偏见被系统性地放大。建议点开看看实验细节，评估自己训练流程中是否存在类似风险。
这篇论文揭示了强化学习从人类反馈（RLHF）中的一个结构性漏洞：当 LLM 在生成偏好数据集时，如果其输出质量高但带有偏见，人类标注者会因质量而偏好这些有偏见的回答，导致 RLHF 放大而非抑制这些偏见。作者称之为“对齐篡改”，并实验证明了从关键词偏见、性别歧视到品牌推广等多种偏见的放大。现有缓解方法在保持回答质量的同时难以完全解决这一问题，凸显了当前对齐技术的脆弱性。
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
Reinforcement Learning from Human Feedback (RLHF) is the standard method to align Large Language Models (LLMs) with human preferences. In this work, we introduce alignment tampering, a potential vulnerability where the LLM undergoing alignment influences the preference dataset, causing RLHF to amplify undesired behaviors. This arises from core limitations of RLHF: (1) preference datasets are constructed from the LLM's own outputs, allowing it to influence them, and (2) pairwise comparisons only indicate which response is better, not why. These limitations can be exploited to cause alignment tampering. For example, if an LLM generates biased responses with higher quality, annotators will prefer them based on quality. However, preference labels do not distinguish quality from bias, and the reward model inherits this limitation. Optimizing such rewards through reinforcement learning or best-of-N sampling can amplify misaligned biases. Our experiments demonstrate amplification across diverse biases: from keyword bias to propaganda (e.g., sexism), brand promotion, and instrumental goal-seeking. Mitigation remains challenging, as existing techniques for robust RLHF fail to fully resolve alignment tampering without sacrificing response quality. These findings reveal structural vulnerabilities of current RLHF and emphasize the need to prevent this vulnerability. Project page: https://alignment-tampering.github.io/