研究复现:RL 训练使 LLM 红队攻击多样性收窄
Attacker Diversity Reduction in Adaptive LLM Red Teaming
对做 LLM 安全的人来说这篇挺有用:它用 Qwen2.5-14B 攻 Phi-4-mini 的实战设置复现了一个此前只有 GRPO 报告过的现象,还给出了 ADR 这个可量化的指标。如果你在搭自动化红队流程,这里提供了具体训练配置,可以直接借鉴。
一项复现实验验证了强化学习在提升自动化红队攻击成功率的同时会收窄攻击多样性的现象。实验使用 Qwen2.5-14B 作为攻击方、Phi-4-mini 作为防御方,通过 DPO 训练,在 26 个 AdvBench 行为上评估,三轮独立运行的 Attack Diversity Retention(ADR)分别为 0.63、0.61 和 0.63,均低于 0.75 的多样性收窄判定阈值。
技术细节上,攻击方采用仅成功奖励、长度无偏损失、有界更新、warm-up 与 ratio clipping 等手段,共执行 24 次攻击方和 12 次防御方基于 LORA 的更新,训练在 12 轮后依据开发集 checkpoint 决策停止。第 4 轮和第 8 轮的中间检查未达标,第 12 轮三轮全部达标。
值得注意的是,尽管攻击向量收窄了约 25% 以上,攻击成功率仍随每轮更新持续提升,说明多样性与有效性在对抗训练中并非同步变化。
Attacker Diversity Reduction in Adaptive LLM Red Teaming
Reinforcement learning can make automated red-teaming more successful while narrowing the attacks it produces. Co-training an attacker and a defender with group-relative policy optimization (GRPO) has been reported to reduce attacker diversity. This experiment reproduces attack diversity reduction using three runs with a Qwen2.5-14B attacker and a Phi-4-mini defender trained by direct preference optimization (DPO). The attacker receives a success-only reward and uses a length-unbiased loss, bounded updates, warm-up and ratio clipping. The attacker is evaluated on 26 AdvBench behaviors. Training ended after 12 rounds, following a decision based on development checkpoints, so the reported intervals are nominal. Attack Diversity Retention (ADR) compares the within-behavior of the trained attacker spread with the untrained attacker. In order for an attack diversity to be consider as having been reduced, a criterion requires the ADR to be below a 0.75 target with a 95% confidence interval together with continued attacker effectiveness. Therefore, if an attacker narrows its attack vector by 25% consistently its considered less diverse. After 24 attacker and 12 defender LORA-based updates, ADR for the three runs were 0.63, 0.61 and 0.63, respectively. Each run met the criterion at round 12; the earlier checks at rounds 4 and 8 fell short. Despite attacks being less diverse, attack success continued to improve as the attacker was updated after each training run.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Qwen2.5, DPO, AI safety, AdvBench.
Source attribution: curated from arXiv cs.LG (@Chris Tava). Cross-source corroboration: 3 independent sources have covered this development (including elvis, rohanpaul_ai), confirming industry-wide signal strength.