Geometric-Mean Policy Optimization
Introduces a stabilized alternative to Group Relative Policy Optimization (GRPO), which is widely used to improve reasoning capabilities in large language models via reinforcement learning. GRPO optimizes the arithmetic mean of token-level rewards but suffers from training instability due to extreme importance sampling ratios. GMPO addresses this by instead maximizing the geometric mean of token-level rewards, leading to more stable updates.
Ask this paper
GMPO reduces the impact of outlier tokens by leveraging the geometric mean, which naturally downweights extreme importance-weighted rewards and leads to narrower sampling ratio distributions and lower gradient variance.
The method introduces token-level clipping of importance sampling ratios (rather than sequence-level) and allows for a wider clipping range, encouraging greater policy exploration without sacrificing stability. The chosen range (e⁻⁰.⁴, e⁰.⁴) achieves the best performance tradeoff.
Across five math benchmarks (AIME24, AMC, MATH500, Minerva, OlympiadBench) and one multimodal benchmark (Geometry3K), GMPO outperforms GRPO with a +4.1% Pass@1 gain in reasoning tasks and +1.4% in multimodal reasoning using 7B models.
Theoretical and gradient analysis show that GMPO yields more robust and balanced updates. Empirically, it maintains higher token entropy and smaller KL divergence from the base model, reflecting better exploration and training stability.
Ablation studies confirm the effectiveness of the geometric mean, token-level clipping, and normalization in achieving improved performance and stable training behavior.