DistIL解决了RLVR只利用最终答案信号的局限，让模型能从执行过程和专家反馈中学习，做推理模型和编程助手的团队值得关注这一新范式。
当前主流的强化学习从可验证奖励（RLVR）方法仅使用最终答案正确与否的单一比特信号，忽略了执行轨迹、工具输出、专家修正和模型自评估等丰富反馈。研究者提出DistIL方法，基于分布化DAgger算法，通过前向交叉熵目标利用这些反馈，实现序列级别的信用分配。理论证明该方法能保证单调策略改进和遗憾界，而基于反向KL或JS散度的自蒸馏目标则无法保证。实验表明，DistIL在科学推理、编程和数学问题求解等多个领域优于RLVR和自蒸馏基线。
Reinforcement Learning from Rich Feedback with Distributional DAgger
Reasoning models have advanced rapidly, but the dominant reinforcement learning from verifiable rewards (RLVR) recipe remains surprisingly narrow: sample many responses and reward each with a single bit indicating whether the final answer is correct. Yet many settings provide rich feedback, including execution traces, tool outputs, expert corrections, and model self-evaluations. We study how to use such feedback through a distributional variant of the classic imitation learning algorithm DAgger, where the learner has local access to an expert distribution on states visited by the current policy. This yields a simple forward cross-entropy objective that admits a blackbox expert and whose sequence-level gradient {conduct rich credit assignment by propagating} future expert-student disagreement back to earlier decisions. We show that prior RL with self-distillation objectives based on reverse KL or Jensen-Shannon fail to guarantee monotonic policy improvement: even when the expert has higher reward, their updates may increase probability on worse actions. In contrast, we show that forward cross-entropy admits monotonic policy improvement and enjoys guarantees on regret. We further show that our objective optimizes a lower bound on teacher-weighted likelihood of success, leading to improved Pass@N. Empirically, our approach, DistIL, improves over RLVR and RL with self-distillation baselines across a variety of domains: scientific reasoning, coding, and solving hard mathematical problems.