Cognitive Behaviors that Enable Self-Improving Reasoners
Free while signed in. Answers cite the passages they came from.
Researchers from Stanford University and colleagues investigate why some language models excel in reinforcement learning (RL)-based self-improvement, while others quickly plateau. The study identifies four cognitive behaviors-verification, backtracking, subgoal setting, and backward chaining-that underpin successful problem-solving in both humans and language models. Key findings:
Cognitive behaviors drive model improvement - Models naturally exhibiting verification and backtracking (like Qwen-2.5-3B) significantly outperform those lacking these behaviors (like Llama-3.2-3B) in RL tasks such as the Countdown math game.
Behavior priming boosts performance - Introducing cognitive behaviors into models through priming substantially enhances RL-driven improvements. Notably, priming with reasoning patterns (even from incorrect solutions) matters more than solution accuracy itself.
Pretraining behavior amplification - Curating pretraining data to emphasize cognitive behaviors enables previously lagging models (e.g., Llama-3.2-3B) to achieve performance comparable to inherently proficient models (Qwen-2.5-3B).
Generalization potential - The identified cognitive behaviors, once amplified through training, show generalizable benefits across reasoning tasks beyond the specific Countdown game used in experiments. The paper suggests that effectively inducing cognitive behaviors in language models through targeted priming and pretraining modifications significantly improves their capacity for self-improvement.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack