Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents
Liming Pu and colleagues at Alibaba Research argue that the widely assumed ceiling on outcome-only RL for small open agent models is a practice artifact rather than a property of the method, and present CANOPY, a stripped-down protocol that tops the AppWorld leaderboard with a 14B policy trained purely through environment interaction.
Ask this paper
Two named failure modes. The paper isolates signal starvation, where group-relative RL produces no gradient unless a rollout group mixes wins and losses, and policy drift, where too many updates over too small a task pool collapse the sampling distribution. Both are training-loop pathologies, not reward-density problems.
The fix is subtraction, not addition. CANOPY scales same-task exploration until natural contrast reappears, keeps every update on-policy and KL-anchored, and confines the loss to the agent's own action tokens. No dense rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration.
Numbers that make the claim hard to wave away. Qwen3-14B reaches 86.9 Test-Normal TGC and 67.6 Test-Challenge on AppWorld, topping the public leaderboard in Feb 2026, and the same principles lift Qwen3.5-9B by 16.6 points on SWE-bench Verified.
Test-time budget is part of the recipe. The enlarged interaction budget is cashed in at inference, which means the trained policy is designed around a deployment assumption rather than a fixed rollout length.
Why it matters. If long-horizon competence can be internalized into weights with a minimal loop, a large fraction of current agent scaffolding is compensating for a training bug rather than adding capability.
Abstract
Reinforcement learning is a natural way to post-train LLM agents for long-horizon interactive tasks judged only by end-of-task verification, yet a shared belief holds that outcome-only RL soon hits a ceiling on small open models. Recent work therefore compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration. We argue the ceiling is an artifact of two failures of common practice. Signal starvation: group-relative RL with sparse outcome-only rewards yields a gradient only when a task's rollout group mixes successes and failures, so under-scaled exploration silences exactly the hardest, most instructive tasks. Policy drift: squeezing many updates out of a small task pool degrades the policy itself, as an unanchored objective lets the sampling distribution collapse exactly when saturation has already made informative groups rare. We present CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens, then cash in an enlarged interaction budget at test time. On AppWorld, a long-horizon interactive coding benchmark, a Qwen3-14B policy trained with CANOPY through environment interaction alone--without task-specific supervision, auxiliary credit signals, or elaborate agent scaffolding--topped the public leaderboard (Feb. 2026; Test-Normal TGC 86.9, Test-Challenge 67.6), and the same design principles lift Qwen3.5-9B on SWE-bench Verified by 16.6 points. Agentic RL alone thus internalizes long-horizon capability directly into a small open model; we plan to release the complete training stack at https://github.com/AlibabaResearch/SignalCoverageRL.