Open-Reasoner-Zero
Open-Reasoner-Zero (ORZ) is an open-source large-scale minimalist reinforcement learning (RL) framework that enhances reasoning capabilities. ORZ demonstrates significant scalability requiring only 1/30th of the training steps of DeepSeek-R1-Zero-Qwen-32B to outperform it on GPQA Diamond. Key contributions and findings:
Ask this paper
Minimalist RL Training Works β Unlike traditional RLHF setups, ORZ removes KL regularization and relies on vanilla PPO with GAE (Ξ»=1, Ξ³=1) and a simple rule-based reward function to scale both response length and reasoning accuracy.
Outperforms Closed-Source Models β ORZ-32B beats DeepSeek-R1-Zero-Qwen-32B on GPQA Diamond while using significantly fewer training steps, proving that training efficiency can be drastically improved with a streamlined RL pipeline.
Emergent Reasoning Abilities β ORZ exhibits "step moments", where response lengths and accuracy suddenly increase, indicating emergent reasoning capabilities with continued training.
Massive Scaling Potential β ORZβs response length scaling mirrors trends seen in DeepSeek-R1-Zero (671B MoE), but with 5.8x fewer training steps. Training shows no signs of saturation, hinting at even further gains with continued scaling.
Fully Open-Source β The training code, model weights, data, and hyperparameters are all released, ensuring reproducibility and enabling broader adoption in the research community.
Mathematical & Logical Reasoning β ORZ significantly improves accuracy on benchmarks like MATH500, AIME2024, and AIME2025 with a simple binary reward system that only evaluates answer correctness.
Generalization β Without any instruction tuning, ORZ-32B outperforms Qwen2.5-32B Instruct on MMLU_PRO, showcasing its strong reasoning generalization despite being trained purely on RL.