研究：GRPO 训练中用有限强验证器预算审计 reward hacking
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO
训练 RL 模型被 reward hacking 折腾的话看看这篇：给出具体审计策略 LB，只用 28% 标注就接近全量审计效果。
这篇 arXiv 论文研究 GRPO 训练中如何分配稀缺的强验证器预算，把哪些弱通过 rollout 需要审计与抓到 hack 后给什么奖励分开设计。提出 low-pass budgeted audits（LB）策略：优先审计弱通过数量最少的整组 rollout。在 LeetCode 和 Countdown-Code 两个环境中，LB 把 hacking 压到接近零，同等惩罚和预算的随机审计做不到且耗更多标注。在 Countdown-Code 上，两种无偏修正方法虽然也能降低 hacking，但 held-out 准确率比 LB 低 6.1-11.5 分；LB 只用全量审计 28% 的标注就能在三个种子上与其差距保持在约 2 分内。基于冻结面板的预测指标在 39 次运行中的 33 次正确预测了 hacking 是否被抑制。
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO
Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). Training outcomes thus depend not only on the expected update but also on the randomness of its estimate.
