代码生成中的熵正则化排序掩码策略优化
Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
这篇论文提出ERPO方法，解决了代码生成中TTRL的奖励信号问题，在编程基准测试中大幅提升性能。
研究人员提出probe-driven TTRL方法，通过从问题语句构建无输出探测输入，执行候选程序并定义探测共识奖励(PCR)。ERPO方法将低PCR转化为保守的负更新，通过排序掩码和熵上限控制策略漂移。在编程基准测试中，ERPO显著提升了pass@1和pass@k指标，在领域内适应和零样本转移中均有表现。
Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal. To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus. We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.
