Teaching Arithmetic to Small Transformers
First page
Paper summary
Trains small transformers on chain-of-thought style data for arithmetic with large gains.
Ask this paper
01
Data format matters: Shows that reformulating arithmetic into explicit step-by-step data dramatically improves small-model accuracy and convergence.
02
Emergence from curriculum: Fine-grained reasoning traces enable small transformers to learn multi-digit arithmetic that would otherwise require orders-of-magnitude more scale.
03
High-quality data thesis: Supports the emerging 2023 thesis that instructive, well-formatted data beats brute-force scaling for specific skills.
04
Small-model research: Informed the later Phi-series (Phi-1, Phi-1.5, Phi-2) "textbooks are all you need" data-quality research program.