SmolLM2

SmolLM2 demonstrates that strategic data curation beats scale through a 1.7B parameter model trained on 11 trillion tokens using iterative data mixing optimization. The data-centric approach introduces three specialized datasets (FineMath, Stack-Edu, SmolTalk) and dynamically refines composition across training stages, achieving superior performance over Qwen2.5-1.5B and Llama3.2-1B while enabling practical on-device deployment. - **Data-centric training philosophy**: Instead of extensive hyperparameter tuning, the team manually refined dataset mixing rates at each training stage based on previous performance. This iterative optimization of data composition proves more effective than architectural modifications for small models, demonstrating that "what you train on" matters more than "how many parameters you have." - **Specialized dataset creation**: Developed FineMath for mathematical reasoning, Stack-Edu for educational code examples, and SmolTalk for instruction-following when existing datasets proved inadequate. This targeted dataset engineering addresses specific capability gaps that generic web text cannot fill, enabling comprehensive competence despite compact size. - **Multi-stage training with strategic mixing**: Trained on ~11 trillion tokens combining web text, math, code, and instruction data across multiple stages. Each stage's data mixture is dynamically adjusted based on evaluation results, allowing the training process to self-correct and optimize for balanced capabilities across domains. - **Performance exceeding larger models**: SmolLM2-1.7B outperforms recent competitors like Qwen2.5-1.5B and Llama3.2-1B, validating that strategic data curation compensates effectively for parameter constraints. The model achieves competitive results on reasoning benchmarks while maintaining the efficiency needed for edge deployment. - **Three-size deployment flexibility**: Released in 135M, 360M, and 1.7B parameter variants, enabling deployment across resource-constrained devices from mobile phones to embedded systems. This size flexibility ensures developers can select the optimal capability-efficiency tradeoff for their specific hardware constraints. - **Open training recipes and datasets**: Publicly released the complete training methodology, datasets (FineMath, Stack-Edu, SmolTalk), and model weights. This transparency enables reproducible research into efficient small model development and provides practitioners with production-ready resources for building on-device AI applications.