Textbooks Are All You Need II (phi-1.5)
Free while signed in. Answers cite the passages they came from.
Microsoft's phi-1.5 demonstrates that a 1.3B model trained on "textbook-quality" synthetic data rivals much larger models on reasoning.
Small but capable: A 1.3B parameter model trained on only 30B tokens competes or outperforms much larger open models on reasoning tasks.
Synthetic textbook data: Training data consists of AI-generated "textbook-quality" content, deliberately curated for pedagogical clarity rather than web breadth.
Data quality dominates: Suggests that data quality and pedagogical structure matter more for reasoning emergence than raw parameter count - a provocative counter to pure-scaling narratives.
Phi-family kickoff: Establishes the recipe that the phi-2, phi-3, and phi-4 releases would refine, popularizing synthetic-data-heavy small LLM training.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack