做自蒸馏或强化学习的研究者会发现，步骤对齐反馈比传统奖励信号更高效，直接提升模型推理质量，值得在实验中尝试这种上下文设计。
该研究探讨了自蒸馏中上下文设计的关键作用，通过训练求解器接收冻结批评者的反馈，比较了三种条件：二元奖励、参考解决方案和步骤对齐批评。步骤对齐批评在Avg@12指标上比GRPO高出16.11分，比参考解决方案条件高出5.27分。分析表明，步骤对齐反馈仅针对推理失败的token，保留正确行为，而参考解决方案迫使模型在每个token上改变行为，导致效率降低。研究揭示了反馈与求解器推理的结构对齐是自蒸馏有效性的关键驱动因素。
The Role of Feedback Alignment in Self-Distillation
Conditioning a language model on additional context, such as feedback on a previous attempt, typically improves its response. Self-distillation trains the model to retain this improvement when the context is not present. The method works by matching the model's output distribution under two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns therefore depends on what context the self-teacher receives, yet the design of this context remains largely unexplored. We study context design for self-distillation by training a solver on feedback from a frozen critic. We compare three conditions: (i) a binary reward (GRPO), (ii) the reference solution, and (iii) a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and reference-solution-conditioned self-distillation by 5.27 points (Avg@12). Per-token advantage analysis reveals why: step-aligned feedback targets only the tokens where reasoning fails, leaving correct behavior intact. Conditioning on the reference solution, by contrast, pressures the model to change its behavior at every token (even correct steps) because an alternative derivation inevitably differs in phrasing and approach. This suggests that structural alignment between feedback and the solver's reasoning is a key driver of self-distillation effectiveness.