OPPD Distillation Method Raises Single-Generation Reasoning Accuracy
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution
The non-obvious result is that inference-time candidate scoring can be absorbed into the weights, so the sharpening gain survives a single forward pass at deployment. Teams doing RLHF-style training on math or code can stack OPPD on top of GRPO for up to 9.3 extra points, and unlike GRPO it needs no verified reference answers, which matters where reward data is scarce. The loss coefficient gives direct control over the absorbed sharpening exponent.
Researchers propose on-policy power distillation (OPPD), a training method that lets a model produce sharpened answers in one generation instead of sampling many candidates at inference.
The method pairs the model with a frozen teacher's power distribution via a sequential Monte Carlo sampler and uses the same probabilities in a maximum-likelihood update.
On MATH500 and GSM8K, single-generation accuracy improves by up to 23.0 and 27.
3 points over the untrained model, and one generation beats published power sampling with 64 candidates by 2.4 and 3.
5 points.Compared with GRPO trained with verified rewards, OPPD scores 3.
8, 4.0 and 5.
4 points higher on MATH500, GSM8K and AIME without any reference answers, and applying OPPD after GRPO adds up to 9.
3 points.Code is available at github.
com/ArminAzizi98/OPPD.
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution
A language model can give a correct answer more probability than any single incorrect answer and still usually sample an incorrect one, because the incorrect answers together hold more probability. The power distribution raises each complete answer's probability to a power above one and renormalizes, shifting probability toward answers the model finds most likely (sharpening). Sampling from it improves reasoning without changing parameters, but needs many scored candidates per query. We show that a model can instead be trained to produce such answers in one generation. On-policy power distillation (OPPD) runs a sequential Monte Carlo sampler in which the model being trained generates candidates and a frozen teacher's power distribution weights them; the same probabilities weight each answer in a maximum-likelihood update. Training raises single-generation accuracy by up to 23.0 points on MATH500 and 27.3 on GSM8K over the untrained model at the same temperature, and one generation scores 2.4 and 3.5 points above published power sampling with 64 candidates, recovering 94 percent of the gain that 16 candidates give the untrained model. For context, against GRPO trained with verified rewards from the same checkpoint and budget, OPPD scores 3.8, 4.0 and 5.4 points higher on MATH500, GSM8K and AIME using no reference answers; the two are complementary, and OPPD applied after GRPO adds up to 9.3 points. Trained only on mathematics, OPPD raises HumanEval accuracy by up to 5.3 points. One loss coefficient moves the sharpening exponent the model absorbs between 1.19 and 2.02, against 1.14 for ordinary on-policy distillation, and it rises mostly on the model's own answers. Gains hold across model families and sizes, including a model already trained with verified rewards, where lowering the temperature gives nothing and OPPD adds 4.4 points on MATH500. Code: https://github.com/ArminAzizi98/OPPD.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include OPPD, power sampling, GRPO, distillation.
Source attribution: curated from arXiv cs.AI (@Erfan Baghaei Potraghloo, Seyedarmin Azizi, Arya Fayyazi, Saeid Shokoufa, Mehdi Kamal, Souvik Kundu, Massoud Pedram).