Reinforcement Pre-Training

Core method: At each token position in a text sequence, the model first generates a reasoning trace (chain-of-thought) and then predicts the next token. If the prediction is a valid prefix of the ground-truth continuation, a reward is assigned. Multiple rollouts are used per context, and the model is trained via on-policy RL.

Better than standard pretraining: RPT significantly outperforms standard next-token prediction and chain-of-thought reasoning baselines (without RL), achieving higher accuracy on tokens of varying difficulty and even rivaling larger models in performance. RPT-14B, for instance, matches or exceeds R1-Qwen-32B’s accuracy on the OmniMATH benchmark.

Strong scaling laws: RPT exhibits clean power-law scaling with respect to training compute across difficulty levels, with prediction accuracy consistently improving as compute increases, and fitting closely to theoretical curves.

Improves downstream RL and generalization: Fine-tuning RPT models with reinforcement learning on tasks with verifiable answers (e.g., Skywork-OR1) shows faster and stronger gains compared to models trained with standard objectives. Zero-shot evaluation on SuperGPQA and MMLU-Pro benchmarks reveals that RPT-14B in reasoning mode surpasses R1-Distill-Qwen-32B by a significant margin.

Promotes structured thinking: Analysis of reasoning traces reveals that RPT-14B employs more hypothesis generation, deduction, and reflective patterns compared to traditional problem-solving models, supporting the claim that RPT fosters deeper reasoning habits during training.