Reinforcement Learning via Self-Play

Emergent reasoning via self-play: RLSP trains an LLM on reasoning tasks by having it generate solution steps and reward itself for exploration and correctness, effectively enabling it to search for answers like an algorithm.

Three-phase training: (1) Begin with supervised fine-tuning on human or synthetic reasoning traces, (2) add an exploration reward to encourage trying diverse solution paths, and (3) employ an outcome verifier in RL to ensure answers are correct (preventing reward hacking).

Notable performance gains: On math benchmarks, a relatively small model (8B) fine-tuned with RLSP saw +23% accuracy on MATH dataset, and a 32B model gained +10% on challenging Olympiad problems—significant jumps achieved by training for better reasoning.

Uncovering new behaviors: RLSP-trained models exhibit emergent problem-solving behaviors like backtracking on flawed steps and self-verification of answers. This suggests that appropriately scaling the training process can induce more robust reasoning capabilities in LLMs.