One-Shot OPD Method Reduces Training Data
This research reveals that on-policy distillation's gains come primarily from algorithm improvements rather than data volume, allowing dramatic reduction in training requirements. Developers can implement One-Shot OPD to train models more efficiently, with potential cost savings and faster iteration cycles for AI companies in both China and the US.
One-Shot OPD from TsinghuaNLP and partners reduces training data to a single query while achieving 87% of full-data OPD's gain on math benchmarks.
The method reaches 71.5% state coverage with one query and 98.
9% with 16 diverse queries across Qwen, Llama and OLMo models.Alignment improvement slows similarly regardless of training-set size, from 1 query to 17k queries, showing OPD is algorithm-starved rather than data-overfed.
Post-training pipelines now use on-policy distillation (OPD) to hand a student the teacher's full next-token distribution at every prefix it visits: the student samples its own rollouts, and Qwen3, MiMo, GLM-5, DeepSeek-V4 and Kimi K3 all pair it with SFT and RL. Yet work on OPD has almost all stood on the algorithm side, treating the training set as given—so how much of OPD's gain does the data account for? Introducing One-Shot OPD, from @TsinghuaNLP (OpenBMB member) with the University of Chinese Academy of Sciences, Northeastern University, UIUC and Johns Hopkins University. It cuts the training set to one query, and the answer is that OPD is data-overfed but algorithm-starved.
1️⃣ One query, hundreds of steps. On math it goes from 59.1 to 68.5 by step 300, against 69.8 for full-data OPD—87% of its gain. It holds across code, instruction following and agentic tool use, and across Qwen, Llama and OLMo; a query the student never solves works about as well as one it always solves. 2️⃣ States, not questions. A prefix is a state where the teacher gives a target distribution, so 64 rollouts per step already yield tens of thousands of supervised positions. One query reaches 71.5% state coverage of what full-data OPD visits; 16 diverse queries reach 98.9% and match full data—extra queries buy new states, not new questions. 3️⃣Alignment slows, not the data. The student keeps improving, but each update absorbs less of what is left, which is why a run takes hundreds of steps rather than tens. This decline hardly depends on training-set size: on 1, 4, 16 and all 17k queries, alignment slowed at a similar pace. What limits a run is not how much data it gets, but how fast the student absorbs it.
📄 Paper: https://t.co/KbK6QhYA4t 💻 Code: https://t.co/dlaFpz9hNJ #AI #THUNLP #OpenBMB #LLM #PostTraining #Distillation #OpenSource
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include OPD, Qwen, GLM-5, DeepSeek-V4.
Source attribution: curated from OpenBMB (@OpenBMB).