Score Centering Stabilizes Off-policy RL
Score Centering Stabilizes Off-policy Reinforcement Learning
For researchers working on reinforcement learning, this paper presents a practical method to stabilize RL under training-inference mismatch, which is a common challenge when scaling models.
This paper introduces a method called score centering to address training-inference mismatch in reinforcement learning. It derives an additive correction term that cancels drift between training and inference engines, stabilizing RL. When training models from 0.6B to 30B parameters, score centering matches or outperforms importance-sampling methods under quantization, especially as the mismatch becomes more severe. The correction also composes with importance sampling, improving performance in staleness experiments.
Score Centering Stabilizes Off-policy Reinforcement Learning
Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.