Group Sequence Policy Optimization
Free while signed in. Answers cite the passages they came from.
This paper introduces GSPO, a new RL algorithm designed to improve the training of large language models, particularly under high compute and long-sequence regimes. Unlike GRPO, which applies token-level importance weights, GSPO performs optimization entirely at the sequence level, aligning the unit of reward with the unit of optimization to resolve instability and inefficiency in large-scale RL training.
Core idea: GSPO replaces token-level importance ratios with a sequence-level formulation based on normalized likelihood ratios, avoiding the variance explosion and misaligned updates that plague GRPO during long-sequence training.
Training stability and performance: GSPO eliminates the need for a value model (as in PPO) or token-wise reweighting (as in GRPO), and leads to more stable convergence, even in challenging Mixture-of-Experts (MoE) settings, by clipping entire responses rather than individual tokens. This results in significantly better training efficiency despite higher clipping rates (15% vs 0.13% for GRPO).
No need for Routing Replay: In MoE models, token-level importance ratios fluctuate due to routing volatility. GRPO requires Routing Replay to maintain consistent expert paths across updates. GSPO sidesteps this by relying on the sequence-level likelihood, which is more stable and obviates the need for additional stabilization tricks.
Infrastructure simplicity: Since GSPO only requires sequence-level likelihoods, it is more tolerant of numerical differences between inference and training engines. This allows for greater flexibility in infrastructure (e.g., using inference engine outputs directly), particularly in multi-turn or disaggregated RL settings.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack