Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains
Sumeet Ramesh Motwani, Jakob Foerster, Shital Shah and colleagues at the University of Oxford and Microsoft Research introduce Rubric Curriculum RL (RcRL), which trains on open-ended tasks using the gap between judging and generating, with no new data or stronger teacher.
Ask this paper
Method. RcRL combines pairwise preferences over rollouts with a curriculum over rubric criteria. Pairwise comparison gives a more discriminative signal than absolute scores, and the changing curriculum makes the objective non-stationary.
Training dynamics. Baselines plateau within a few hundred steps. RcRL preserves output entropy and keeps improving past 1,000 steps, with less reward hacking.
Creative writing. RcRL reaches a 70.5% human win rate against the base model and a 73.9% win rate on FictionBench, with gains across benchmarks and judges.
HealthBench. RcRL improves over the base model by 14.6% on the full set and 25.4% on the consensus subset, beating all training baselines including instance-specific rubrics.
Scaling. Improvements grow with model size.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. The generation-verification gap, where judging good outputs is easier than producing them, offers a potential path to improvement. However, rubric-based judges are often imprecise and stationary, so policies can overfit or reward hack them. We introduce Rubric Curriculum RL (RcRL), which exploits this gap without new data, human annotations, or stronger teacher models. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring and a non-stationary objective. Whereas baselines plateau within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL achieves a 70.5% human win rate against the base model, with gains across benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all training baselines, including instance-specific rubrics.