Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
Shu Zhao and Tan Yu (NVIDIA) argue that agentic on-policy distillation fails mainly because next-token scoring is myopic rather than because of compounding error, and propose Rollout-to-Scorer Distillation (R2S), which fine-tunes the teacher's scores before distillation begins.
Ask this paper
Rollout-scorer gap. From the states where the student diverges most, the teacher's own takeovers still complete the task, yet a student trained on the teacher's token scores keeps running out of turns. Per-token scores do not penalize safe, delaying moves that an outcome reward would score as zero.
Repair procedure. R2S mines high-disagreement states from the untrained student, runs teacher takeovers from them, keeps states where enough takeovers pass a reward-free check such as consensus or an LLM judge, and fine-tunes the teacher to give takeover-informed scores from the divergence turn onward without the takeover in context.
No task reward. The repaired teacher is frozen and serves as the only scorer for standard OPD. No task reward is used during repair or training.
Results. Across three environments, two teachers and four students, R2S beats three state-of-the-art agentic OPD methods in all 12 teacher-student-environment cells, by 12.8 success points on average over the best of them in each cell.
Abstract
Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree. We show this explanation is incomplete. From the states where the student departs most from the teacher, the teacher's own takeovers still finish the task, yet the student trained on its token scores keeps running out of turns. We attribute this to the myopia of next-token scoring. The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks. We call this difference between what the teacher does and what it scores the rollout-scorer gap. We propose Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation by mining high-disagreement states from the untrained student, running teacher takeovers from them, keeping the states where enough takeovers pass a reward-free check such as consensus or an LLM judge, and fine-tuning the teacher to give its takeover-informed scores on every assistant token from the divergence turn onward, without the takeover in context. The repaired teacher is frozen as the sole scorer for standard OPD, requiring no task reward during repair or training. Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher-student-environment cells, by 12.8 success points on average over the best of them in each cell. Project page: https://tree-shu-zhao.github.io/r2s/.