Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
Xi Chen, Ruixue Ding, Pengjun Xie, Hui Xiong and colleagues (Alibaba Group, HKUST Guangzhou) introduce Rollout-Time Intervention (RTI), which uses evaluation rubrics during rollout to repair failed agent trajectories so that RL has successes to learn from on tasks the policy rarely solves.
Ask this paper
On-policy bottleneck. When a task is beyond the current policy, successful rollouts are too rare to provide gradient and agent RL training often collapses.
Validator, Corrector, Verifier. A multi-agent pipeline reads reward-side evidence such as rubrics, finds the error in a failed rollout, rewrites it into a successful trajectory and checks that no privileged information leaked into the policy context.
Off-policy control. Perplexity-based gating drops intervened trajectories that drift too far from the policy distribution, and token reweighting concentrates updates on the tokens where the intervention changed the strategy.
Results. With Qwen3-8B, RTI beats GRPO by 24.56 points on tau2-Telecom, 22.00 on Navigation and 27.50 on Claw-Eval-Hard20, where it reaches 50.00% against 26.25% for OPD and 30.00% for LUFFY.
Beyond the base model. On the Claw-Eval-Hard20 subset, Qwen3.5-9B rises from 26.25% to 85.00% Pass@1, 23.75 points above LUFFY, the strongest competing method.
Abstract
Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model's current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at https://github.com/Alibaba-NLP/qqr.