On the Slow Death of Scaling

This essay by Sara Hooker challenges the decade-long assumption that scaling compute always leads to better AI performance. It argues that the relationship between training compute and performance is highly uncertain and rapidly changing, with smaller models now routinely outperforming much larger ones. - **Diminishing returns of scale:** Smaller models like Llama-3 8B and Aya 23 8B now outperform far larger models like Falcon 180B and BLOOM 176B despite having only a fraction of the parameters. This trend is systematic, not isolated. - **Algorithmic improvements matter more:** Progress has been driven by instruction finetuning, model distillation, chain-of-thought reasoning, preference training, and retrieval augmented generation - techniques that add little training compute but yield significant performance gains. - **Scaling laws have limits:** Scaling laws only reliably predict pre-training test loss, not downstream task performance. Many capabilities display irregular scaling curves, and small sample sizes make predictions statistically weak. - **New optimization spaces:** Future progress will come from inference-time compute, malleable synthetic data that can be optimized on-the-fly, and better human-AI interfaces rather than simply adding more parameters. - **Cultural implications:** The belief in scaling has marginalized academia, concentrated breakthroughs in wealthy regions, and led industry labs to stop publishing - reshaping the entire culture of AI research.