做AI安全对齐的团队终于有了一个稳定的GRPO攻防协同训练方案，能同时提升攻击发现能力和防御鲁棒性，建议做红队测试的开发者直接参考。
本文提出AdvGRPO框架，解决了GRPO在攻防协同训练中不稳定的问题。通过密集多通道奖励和分离优势归一化，使攻击者和防御者模型交替更新，从单轮攻击逐步过渡到多轮闭环攻击。实验表明，该方法能生成高效且可迁移的攻击，协同训练的防御者在安全基准上优于基线。这项工作为语言模型的安全对齐提供了新的自适应红队方法。
Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO
AI red teaming must continually adapt to evolving attackers and defenders. Reinforcement learning offers a promising approach to discovering novel attacks, and co-training methods can produce more robust defenders in tandem. Recent works have demonstrated the efficacy of attacker-defender co-training by applying PPO and DPO, but report that GRPO is unstable in this setting. We introduce AdvGRPO, a co-training framework that makes GRPO viable for joint attacker-defender optimization using dense multi-channel rewards and decoupled advantage normalization. Training progresses through a curriculum from single-turn to closed-loop multi-turn attacks before bootstrapping co-training, where attacker and defender models are updated in alternation. We show that our method can produce highly effective and transferable attacks and that co-trained defenders outperform baselines on safety benchmarks.