Reinforcement Learning · Training

GDPO

Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, et al.

Free while signed in. Answers cite the passages they came from.

Figure 1

The curator’s take

GDPO addresses a critical flaw in training language models with multiple competing objectives. The authors discover that when applying Group Relative Policy Optimization (GRPO) to multi-reward settings, normalizing distinct rollout reward combinations causes them to collapse into identical advantage values, degrading training signal quality and stability. - **Fundamental flaw identified:** Standard GRPO normalizes rewards across all objectives together, which causes distinct reward combinations to collapse into nearly identical advantage values—destroying the nuanced signal needed for multi-objective optimization. - **Decoupled normalization:** GDPO decouples the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization across competing objectives. - **Consistent improvements:** GDPO demonstrated gains over GRPO across three domains: tool calling, mathematical reasoning, and code generation, improving both correctness metrics (accuracy, defect rates) and constraint adherence (format compliance, output length). - **Practical multi-objective training:** The approach enables training models that must simultaneously optimize for multiple goals—such as being accurate while following format constraints—without the objectives interfering destructively. - **Drop-in replacement:** GDPO can serve as a drop-in replacement for GRPO in multi-reward RL pipelines, requiring minimal changes to existing training infrastructure while providing more stable and effective optimization.

Key points

Fundamental flaw identified: Standard GRPO normalizes rewards across all objectives together, which causes distinct reward combinations to collapse into nearly identical advantage values—destroying the nuanced signal needed for multi-objective optimization.

Decoupled normalization: GDPO decouples the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization across competing objectives.

Consistent improvements: GDPO demonstrated gains over GRPO across three domains: tool calling, mathematical reasoning, and code generation, improving both correctness metrics (accuracy, defect rates) and constraint adherence (format compliance, output length).

Practical multi-objective training: The approach enables training models that must simultaneously optimize for multiple goals, such as being accurate while following format constraints, without the objectives interfering destructively.

Drop-in replacement: GDPO can serve as a drop-in replacement for GRPO in multi-reward RL pipelines, requiring minimal changes to existing training infrastructure while providing more stable and effective optimization.

Abstract

As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.

Every Monday

Get next week’s papers.

The same picks and the same summaries, in your inbox. Free, and 176 issues deep.

Subscribe on Substack