Spurious Advantage Hidden in GRPO
Jiamian Wang and colleagues identify spurious advantage in GRPO, where a rollout that lands on the right answer by guessing receives the same high advantage magnitude as one that reasoned its way there, and propose SIGNBALANCE to remove it.
Ask this paper
Three cases where it occurs: bounded-answer tasks with a small candidate set, open-answer sets that contain bounded sub-cases, and search agents whose budget opens many paths to the same answer.
The formula cannot tell the two apart: within-group reward statistics assign magnitude from the outcome, and a guess and a derivation share the same surface, so the policy is nudged toward guess-like behavior.
Composition-free magnitude: SIGNBALANCE keeps the verifier's sign, applies a global scale, and restores zero-mean balance with a stop-gradient per-class rescaling.
Matches where GRPO is fine, improves where it is not: parity on open-answer math, gains on bounded-answer math and on search agents across scales.
Why it matters: search agents are exactly the bounded-path case, so anyone doing RL on a tool-using agent is likely training in some amount of guessing right now.
Abstract
Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.