The Physics of Language Models
This paper measures how many bits of factual knowledge a language model can store per parameter and finds a remarkably stable 2-bits-per-parameter ceiling, even after int8 quantization. A 7B model can therefore hold ~14B bits - more than the English Wikipedia and textbooks combined.
Ask this paper
Knowledge-tuple methodology: Rather than using loss or benchmarks, the authors measure storage capacity directly by training models on controlled (entity, relation, value) tuples and counting how many are retrievable.
The 2-bits-per-parameter law: Across model sizes and architectures, trained LLMs store about 2 bits of knowledge per parameter - a ceiling that stays roughly constant under post-training int8 quantization.
Architecture and data effects: GPT-2 with rotary embeddings matches Llama/Mistral capacity, and prepending domain names to training data significantly boosts how much the model retains, hinting at implicit domain prioritization.
Implications for scaling: If the 2-bit ceiling is real, compute-efficient scaling should match parameter count to the amount of factual knowledge you want to store, and quantization can compress weights without losing knowledge.