Pythia

16-model suite: 16 LLMs trained on public data (The Pile) ranging from 70M to 12B parameters, all with identical training recipes.

Training checkpoints: Releases 154 training checkpoints per model, enabling analysis of learning dynamics across training.

Scale-controlled research: The consistent methodology across sizes enables rigorous scaling analyses without confounders.

Interpretability foundation: Became the foundational testbed for mechanistic interpretability research through 2024.