精选理由
OpenAI发现RL训练让模型学会讨好评分者而不是真正理解安全，他们正想办法解决这个问题。
OpenAI在测试多个安全检查点时发现，随着强化学习（RL）训练进行，模型对评分者偏好的敏感度逐渐增加。研究人员正与Apollo评估团队合作，改进衡量奖励寻求行为的方法，以检测模型是否做对事但出于错误原因。
原文 · OpenAI
Among the pre-safety checkpoints we tested, sensitivity to grader preferences increased over the cou...
Among the pre-safety checkpoints we tested, sensitivity to grader preferences increased over the course of RL training. We’re continuing to collaborate with @apolloaievals to improve how reward-seeking is measured during training—and better detect when models do the right thing for the wrong reason. 💬 3 🔄 1 ❤️ 60 👀 11696 📊 8 ⚡