OpenELM
Free while signed in. Answers cite the passages they came from.
Apple's OpenELM is a fully-open small language model family (270M, 450M, 1.1B, 3B) that uses layer-wise parameter scaling instead of uniform layer widths. At ~1B parameters it improves on OLMo by 2.36% accuracy while using half the pre-training tokens.
Layer-wise scaling: Rather than allocating parameters uniformly across transformer layers, OpenELM adjusts the width of each layer to place capacity where it matters most for downstream performance.
Efficient training: The ~1B variant achieves 2.36% higher accuracy than OLMo while requiring 2x fewer pre-training tokens, showing that architectural choices can match or beat raw data scale for small models.
Complete open release: Apple ships not just weights but training logs, intermediate checkpoints, pre-training configs, and MLX inference code - a far more complete release than most commercial labs offer.
Reproducibility focus: The release explicitly targets open research, giving the community the artifacts needed to re-trace and audit every step of the training pipeline.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack