这个新方法把强化学习和知识蒸馏结合在一起，解决了RL和OPD各自的短板，跨家族蒸馏效果尤其好。
Distilled RL将教师模型监督集成到强化学习目标中，提供细粒度指导并选择性传递新知识。它包含三个组件：逆重要性采样与剪切、负样本重置、序列级几何归一化。实验表明，在家族内和跨家族蒸馏场景下，Distilled RL在pass@1和pass@k上均显著优于标准RL和基于策略的蒸馏OPD。该方法能有效传递教师模型先前不可用的知识，代码已开源。
Distilled Reinforcement Learning for LLM Post-training
Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.