Scaling Laws for Generative Reward Models
Angel Raychev, Nathan Lile and Rafael Rafailov (SynthLabs, Stanford, Thinking Machines) study generative reward models as online preference providers for creative writing and find that the judge with the best static accuracy is often not the best rewarder.
Ask this paper
Setup. Qwen3 reward models from 0.6B to 14B, trained either as Thinking judges with GRPO or Answer-Only judges with SFT, provide preferences for online DPO, and policies are scored by a fixed learned 32B Gold evaluator.
Reversal. Thinking judges have higher static Gold-label accuracy, but Answer-Only rewarders give higher peak and top-five-average policy Elo in seven of eight matched settings.
Checkpoints. Intermediate Thinking-judge checkpoints can train better policies than later checkpoints with higher static accuracy.
Scaling. Sweeps over rewarder size, policy size, training length and DPO regularization show clear benefits from reward model capacity, including judges larger than the policy. The authors note the ranking is conditional on their learned reference.
Abstract
We study generative reward models (GenRMs) as online preference providers for policy optimization in creative writing. Across Qwen3 reward-model sizes from 0.6B to 14B, Thinking judges trained with GRPO have higher static Gold-label accuracy than Answer-Only judges trained with SFT. This ranking often reverses when the judges train policies through online DPO: Answer-Only rewarders yield higher peak and top-five-average policy Elo in seven of eight matched settings under a fixed, learned 32B Gold evaluator. Intermediate Thinking-judge checkpoints can also outperform later, more statically accurate checkpoints as rewarders. Sweeps over rewarder size, policy size, training duration and DPO regularization show substantial benefits from reward-model capacity, including judges larger than the policy. We analyze trajectories with evaluator-specific Elo arenas, conditional match-resampling intervals and descriptive scaling fits. These results demonstrate an evaluator-rewarder gap in the studied pipelines: fixed-distribution judging accuracy does not determine downstream optimization quality. The comparison changes judge-training objective as well as response format, and the main ranking is conditional on its learned reference rather than a universal human-quality result. A limited follow-up using a different learned judge is inconclusive overall.