R2S Fixes Myopic Teacher Scoring in Agentic On-Policy Distillation
Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
The non-obvious insight is that the failure of agentic distillation comes from the scorer's myopia, not from students drifting out of the teacher's reliable region, so the fix targets the teacher instead of constraining rollouts. Builders training agents via distillation can adopt R2S without task reward labels, using consensus or an LLM judge as the check. Code and details are on the project page at tree-shu-zhao.github.io/r2s/.
A paper on arXiv identifies the rollout-scorer gap in agentic on-policy distillation: a teacher scores each token without seeing the episode's end, so it rewards safe, delaying moves that outcome-based RL would punish. The proposed method, Rollout-to-Scorer Distillation (R2S), repairs the teacher first by mining high-disagreement states from the untrained student, running teacher takeovers, keeping states that pass a reward-free check such as consensus or an LLM judge, and fine-tuning the teacher's scores. The repaired teacher is then frozen as the sole scorer for standard OPD, needing no task reward. Across three environments, two teachers and four students, R2S beats three state-of-the-art agentic-OPD methods in all twelve cells, by an average of 12.8 success points over the best baseline per cell.
Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree. We show this explanation is incomplete. From the states where the student departs most from the teacher, the teacher's own takeovers still finish the task, yet the student trained on its token scores keeps running out of turns. We attribute this to the myopia of next-token scoring. The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks. We call this difference between what the teacher does and what it scores the rollout-scorer gap. We propose Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation by mining high-disagreement states from the untrained student, running teacher takeovers from them, keeping the states where enough takeovers pass a reward-free check such as consensus or an LLM judge, and fine-tuning the teacher to give its takeover-informed scores on every assistant token from the divergence turn onward, without the takeover in context. The repaired teacher is frozen as the sole scorer for standard OPD, requiring no task reward during repair or training. Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher-student-environment cells, by 12.8 success points on average over the best of them in each cell. Project page: https://tree-shu-zhao.github.io/r2s/.
