JustRL
Free while signed in. Answers cite the passages they came from.
JustRL challenges the assumption that complex RL pipelines are necessary for training small language models. Using single-stage training with fixed hyperparameters, the authors achieve state-of-the-art math reasoning performance on two 1.5B models while using 2x less compute than sophisticated multi-stage approaches. - **Simplicity wins:** The recipe uses GRPO with binary rewards, no curriculum learning, no dynamic hyperparameters, no length penalties, and no multi-stage training. The same fixed hyperparameters work across both DeepSeek-R1-Distill-Qwen-1.5B and OpenMath-Nemotron-1.5B without tuning. - **Strong results with less compute:** JustRL-DeepSeek-1.5B achieves 54.9% average across nine math benchmarks, beating ProRL-V2's 53.1% while using half the compute. JustRL-Nemotron-1.5B reaches 64.3%, slightly outperforming QuestA's curriculum learning approach. - **Stable training dynamics:** Training shows smooth, monotonic improvement over 4,000+ steps without the collapses, plateaus, or oscillations that typically motivate complex interventions. Policy entropy stays healthy between 1.0-1.6, and response length naturally compresses without explicit penalties. - **Adding tricks hurts performance:** Ablations reveal that standard optimizations like explicit length penalties and robust verifiers actually degrade results by collapsing exploration. Length penalties dropped AIME24 performance from 55% to 50%, and adding both modifications dropped it to 45%.
Simplicity wins: The recipe uses GRPO with binary rewards, no curriculum learning, no dynamic hyperparameters, no length penalties, and no multi-stage training. The same fixed hyperparameters work across both DeepSeek-R1-Distill-Qwen-1.5B and OpenMath-Nemotron-1.5B without tuning.
Strong results with less compute: JustRL-DeepSeek-1.5B achieves 54.9% average across nine math benchmarks, beating ProRL-V2’s 53.1% while using half the compute. JustRL-Nemotron-1.5B reaches 64.3%, slightly outperforming QuestA’s curriculum learning approach.
Stable training dynamics: Training shows smooth, monotonic improvement over 4,000+ steps without the collapses, plateaus, or oscillations that typically motivate complex interventions. Policy entropy stays healthy between 1.0 and 1.6, and response length naturally compresses without explicit penalties.
Adding tricks hurts performance: Ablations reveal that standard optimizations like explicit length penalties and robust verifiers actually degrade results by collapsing exploration. Length penalties dropped AIME24 performance from 55% to 50%, and adding both modifications dropped it to 45%.
Abstract
Recent advances in reinforcement learning for large language models have converged on increasing complexity: multi-stage training pipelines, dynamic hyperparameter schedules, and curriculum learning strategies. This raises a fundamental question: \textbf{Is this complexity necessary?} We present \textbf{JustRL}, a minimal approach using single-stage training with fixed hyperparameters that achieves state-of-the-art performance on two 1.5B reasoning models (54.9\% and 64.3\% average accuracy across nine mathematical benchmarks) while using 2$\times$ less compute than sophisticated approaches. The same hyperparameters transfer across both models without tuning, and training exhibits smooth, monotonic improvement over 4,000+ steps without the collapses or plateaus that typically motivate interventions. Critically, ablations reveal that adding ``standard tricks'' like explicit length penalties and robust verifiers may degrade performance by collapsing exploration. These results suggest that the field may be adding complexity to solve problems that disappear with a stable, scaled-up baseline. We release our models and code to establish a simple, validated baseline for the community.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack