NOPO：只用负样本做强化学习，应对单侧可验证奖励场景
Negative-Only Policy Optimization for One-Sided Verifiable Rewards
做 RLVR 训练的同学可以看看：当验证器只能判错不能判对时，这篇论文的 NOPO 干脆只用负样本更新策略，避免奖励投机解法。在代码、数学、逻辑基准上 pass@k 都跑赢了标准 RLVR 基线。
arXiv 论文（编号 2611.17287）提出 Negative-Only Policy Optimization（NOPO），针对一类被作者定义为单侧可验证性（OSV）的场景：验证器只能可靠地判定错误答案，而通过的候选答案中可能混有大量假阳性。论文指出，标准 RLVR 在此场景下会直接强化那些钻过部分检查空子的错误解法。
NOPO 的做法是把验证通过的样本当作无标签数据，仅通过抑制可信的验证负样本来更新策略。方法由两部分组成：一个相对参考模型的负样本抑制损失，以及一个组级适配器，按未被判错响应的比例缩放每次更新。
在单元测试、约束检查和 LLM 自验证三类验证器下，NOPO 在代码、逻辑、数学基准上于无 oracle 奖励时 pass@k 超过基线，且增益在推理时扩展和跨基准迁移中保持。
Negative-Only Policy Optimization for One-Sided Verifiable Rewards
Reinforcement learning with verifiable rewards (RLVR) typically relies on a verifier that can reliably judge whether sampled solutions are correct. However, in many realistic settings, complete verification is generally unavailable, while cheap checks can only falsify failures. We formalize this regime as one-sided verifiability (OSV): verified negatives are reliable counterexamples, but verified positives are merely non-falsified candidates and may contain many false positives. This asymmetry makes standard RLVR unreliable, because reinforcing verified positives can directly reward shortcuts that pass partial checks. To address this issue, we propose Negative-Only Policy Optimization (NOPO), which treats verified positives as unlabeled and updates the policy only by suppressing trusted verified negatives. Specifically, NOPO combines a reference-relative negative suppression loss with a group-level adapter that scales each update by the proportion of non-falsified responses. Across unit-test, constraint-check, and LLM self-verifiers, NOPO improves pass@k over baselines on code, logic, and math benchmarks when oracle rewards are unavailable. These gains persist under inference-time scaling and cross-benchmark transfer.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include NOPO, RLVR, 强化学习, 论文.
Source attribution: curated from arXiv cs.LG (@Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, Bo An).