Nathan Lambert Adds Score-Centering to the RLHF Book
Score-centering moving from a niche trick to an industry default signals how RL post-training recipes are converging, and the book chapter offers a derivation that clarifies why training-inference mismatch hurts policy optimisation. Practitioners implementing GRPO-style or StableReinforce-style pipelines can use it to stabilise advantage estimates. It pairs well with the book's existing chapters on reward modelling and policy gradients.
Nathan Lambert, author of the RLHF Book, announced that score-centering is the latest addition to the book. He notes the technique is popular in industry, citing Thinking Machines' Tinker and its StableReinforce method as an example of its use. He describes score-centering as a useful derivation for understanding training-inference mismatch in reinforcement learning. A video version of the explanation is planned.
Have heard from many folks that score-centering is popular in industry (e.g. Tinker's StableReinforce using it), so it's the latest addition to the rlhf book. It's a nice derivation to understand training-inference-mismatch.
Video version explaining it soon.
