Reinforcement Learning via Self-Play
Researchers propose Reinforcement Learning via Self-Play (RLSP) as a framework to train LLMs to “think” through complex problems. Key ideas include:
Ask this paper
Emergent reasoning via self-play: RLSP trains an LLM on reasoning tasks by having it generate solution steps and reward itself for exploration and correctness, effectively enabling it to search for answers like an algorithm.
Three-phase training: (1) Begin with supervised fine-tuning on human or synthetic reasoning traces, (2) add an exploration reward to encourage trying diverse solution paths, and (3) employ an outcome verifier in RL to ensure answers are correct (preventing reward hacking).
Notable performance gains: On math benchmarks, a relatively small model (8B) fine-tuned with RLSP saw +23% accuracy on MATH dataset, and a 32B model gained +10% on challenging Olympiad problems—significant jumps achieved by training for better reasoning.
Uncovering new behaviors: RLSP-trained models exhibit emergent problem-solving behaviors like backtracking on flawed steps and self-verification of answers. This suggests that appropriately scaling the training process can induce more robust reasoning capabilities in LLMs.