More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
Linhao Zhong, Bozhen Fang, Chunhua Shen and colleagues (Zhejiang University, Ant Group) rerun ten self-distillation methods for agentic RL against GRPO under one protocol on ALFWorld, ScienceWorld, Search-QA and WebShop at three model sizes.
Ask this paper
Scale. 132 training runs and 12,124.4 recorded H800 GPU-hours, with evaluation over eight rollout seeds for every backbone and trained method.
Finding 1. Gains over GRPO are limited in scope. No method consistently beats GRPO across settings.
Finding 2. Strong final scores can hide unstable training.
Finding 3. Additional training cost does not consistently produce better performance.
Why it matters. Earlier comparisons reused published baseline numbers from different setups, which made it unclear where reported gains came from.
Abstract
Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published baseline scores under differing experimental conditions, provide few direct comparisons among contemporaneous methods, and give limited attention to training stability and computational cost. These limitations obscure the source of reported gains and the practical strengths and weaknesses of competing methods. Costly agent training further motivates a common basis for comparison. To address these gaps, we compare ten representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop across three model sizes. Our study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. Beyond final task performance, we examine training progress, stability, and measured computational cost. Our evaluation yields three key findings. First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance. We establish this benchmark to support fair, comprehensive comparison and guide future research toward reliable and cost-effective improvements in agent training. Project page: https://aim-uofa.github.io/Self-Distillation-AgentRL/