重新思考奖励监督：Rubric-Conditioned Self-Distillation
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
想提升推理模型训练效果？这篇用评分标准做细粒度自蒸馏，比GRPO和OPSD都强，实验扎实。
提出Rubric-Conditioned Self-Distillation框架，用评分标准替代标量奖励，提供token级指导。
方法分两步：先学习生成任务级评分标准，再训练评分标准引导的推理器。在多个科学推理基准上平均超越GRPO 1.
0分、OPSD 0.9分。
避免了单一参考推理链的噪声和标量奖励的模糊性。
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose \textbf{Rubric-Conditioned Self-Distillation}, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.
This report is categorized under AITOP's "Models" desk. Key entities and technical topics under continuous tracking include Rubric-Conditioned Self-Distillation, 推理模型, 自我蒸馏, 评分标准.
Source attribution: curated from arXiv cs.AI (@Siyi Gu, Jialin Chen, Sophia Zhou, Arman Cohan, Rex Ying).
