Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO
Hyun Bin Park and Du-Seong Chang isolate replay in GRPO down to a single primitive with two decisions, Headroom for what is still worth learning from and Drift for what is still compatible with the current policy.
Ask this paper
A deliberately narrow question: existing replay methods bundle exploration, experience restructuring and mixed-policy optimization, so replay's own contribution is unmeasurable. This strips it to one primitive.
Two orthogonal decisions: Headroom ranks stored groups by remaining learning value, Drift gates them by policy compatibility, and the fresh on-policy stream is left untouched.
No auxiliary machinery: no extra generation, no additional training components, which is what makes the ablation clean.
Matches or exceeds broader replay methods on Avg Mean@32 across mathematical reasoning, multimodal reasoning and Agentic Search.
Where it pays most: in Agentic Search, where environment interaction dominates wall-clock cost, it delivers comparable quality at materially lower time, which is exactly the agentic RL bottleneck.
Abstract
RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay's own contribution difficult to isolate. We ask a focused question: how far can principled replay selection alone go? We introduce Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality at materially lower wall-clock time.