Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Youling Huang, Lin Lin and colleagues from Kuaishou with DUT, XJTU, Tsinghua and other universities show that on-policy distillation helps agentic RL only while the teacher is ahead of the student, and propose GATS, which scales the distillation term by the measured teacher-student gap and drops it at crossover.
Ask this paper
Finding. The gain from on-policy distillation rises monotonically with the teacher-student performance gap and changes sign once the student catches up, at which point the teacher's token-level guidance slows the student down.
GATS. The RL objective gets an on-policy distillation term whose weight is a monotone function of the gap, and the teacher is permanently withdrawn when the smoothed student success rate reaches the teacher's reference.
Smaller teachers work. Because guidance is only needed early, the teacher can be a task-trained model smaller than the student, which lowers distillation cost, and students end up exceeding those teachers.
Results. On ALFWorld, WebShop and ScienceWorld with three Qwen2.5 teacher-student pairs, GATS has the highest average success in all three configurations, 4.37 to 11.87 points above reward-only GRPO at matched rollout budgets. Ablations attribute the gain to the adaptive schedule rather than to distillation alone.
Abstract
Reinforcement learning for long-horizon agents typically relies on sparse outcome-based rewards. This leads to a severe cold-start problem, as early-stage policies often fail to solve sampled tasks, leaving little useful reward signal for learning. To mitigate this problem, we use on-policy distillation (OPD) to provide token-level guidance on the student's own rollouts. We find that the benefit of this guidance depends on the performance gap between the teacher and the student. When the teacher substantially outperforms the student, distillation helps guide the student through the early training stage where outcome rewards provide little learning signal. As the gap narrows and eventually reverses, however, continued distillation becomes less beneficial and may hinder further improvement. Motivated by this observation, we propose Gap-Adaptive Teacher Scheduling (GATS), which augments the student's RL objective with an OPD term whose weight adapts to the teacher-student performance gap. Specifically, GATS gradually reduces teacher guidance as the student approaches the teacher's reference performance and withdraws it once that reference is reached. This enables GATS to leverage task-trained teachers smaller than the student, since teacher guidance is primarily needed during early training. Across ALFWorld, WebShop, and ScienceWorld with three Qwen2.5 teacher-student configurations, GATS achieves the highest average success rate among the compared methods in all three configurations, improving over reward-only GRPO by 4.37%-11.87% under matched student rollout budgets. Code is available at https://github.com/Ricardo-H/guide-then-let-go.