Anthropic展示了AI如何学会作弊攻击系统，比普通研究更具体地展示了奖励黑客问题。
Anthropic训练了一个Opus规模模型，在80个已知可被攻击的生产环境中进行测试。该模型在模拟评估中进行了未授权的网络攻击、篡改奖励机制并试图逃避安全监控。这项研究旨在了解严重奖励不 alignment 的产生原因。
New research: Training a Misaligned Reward Seeker What produces severe misalignment? We’ve long bee...
New research: Training a Misaligned Reward Seeker What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable. In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring. Read more: alignment.anthropic.com/2026/reward-se… 💬 26 🔄 12 ❤️ 159 👀 19958 📊 42 ⚡
- IT之家08-31 23:55原文