ERPO for Code Generation TTRL
Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
ERPO improves code generation reliability without labeled test data, making it valuable for developers in resource-constrained environments.
The paper introduces probe-driven test-time reinforcement learning for code generation, addressing the challenge of comparing programs by surface form. It proposes Probe Consensus Reward (PCR) as a behavioral training signal for open-vocabulary programs. The authors then develop Entropy-Regularized Rank-Masked Policy Optimization (ERPO) to handle reward hacking through spurious consensus. ERPO improves pass@1 and pass@k metrics on coding benchmarks for both in-domain adaptation and zero-shot transfer.
Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal. To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus. We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.
