OPD 蒸馏方法让低比特量化模型恢复推理能力
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
量化到 2-bit 后模型做数学题会发疯循环？这篇论文用 on-policy 蒸馏把 MATH-500 保留率从 35% 拉到 70%，做端侧部署的可以看看。
论文针对低于 3-bit 量化导致数学和代码推理能力大幅下降的问题，提出 on-policy distillation（OPD）训练阶段，让学生在量化推理路径上生成，并由冻结的全精度教师在其自身前缀上给出反馈。在 4 个模型上以 2.79 和 1.88 有效比特测试，OPD 将 MATH-500 的 BF16 性能保持率从 35% 提到 70%，HumanEval 从 66% 提到 91%。该方法结合量化感知蒸馏（QAD）的低比特初始化与 OPD 的推理恢复，解决长生成退化成重复循环的问题。
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
Quantization-aware distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathematical and code reasoning substantially impaired. Long generations often degenerate into repetitive loops, exhausting the decoding budget without completing a solution. We trace this gap to quantization-amplified exposure bias: QAD trains on fixed corpus prefixes, while quantization-induced deviations compound along the model's own autoregressive trajectories. To address this mismatch, we introduce an on-policy distillation (OPD) stage that places teacher supervision where the quantized model actually goes. Starting from a QAD checkpoint, the student generates through the quantized forward path used at deployment and receives feedback from a frozen full-precision teacher on its own prefixes, combining dense token-level guidance with task-verifier rewards. Across four models at 2.79 and 1.88 effective bits, OPD raises average BF16 performance retention from 35% to 70% on MATH-500 and from 66% to 91% on HumanEval while preserving short-form performance, with reasoning gains substantially exceeding those of continued teacher-forced QAD in matched-budget comparisons. By coupling QAD's stable low-bit initialization with OPD's on-policy reasoning recovery, our framework provides a comprehensive sub-3-bit solution that preserves broad capabilities while restoring long-form reasoning.