The Physics of Language Models
Free while signed in. Answers cite the passages they came from.
This paper measures how many bits of factual knowledge a language model can store per parameter and finds a remarkably stable 2-bits-per-parameter ceiling, even after int8 quantization. A 7B model can therefore hold ~14B bits - more than the English Wikipedia and textbooks combined.
Knowledge-tuple methodology: Rather than using loss or benchmarks, the authors measure storage capacity directly by training models on controlled (entity, relation, value) tuples and counting how many are retrievable.
The 2-bits-per-parameter law: Across model sizes and architectures, trained LLMs store about 2 bits of knowledge per parameter - a ceiling that stays roughly constant under post-training int8 quantization.
Architecture and data effects: GPT-2 with rotary embeddings matches Llama/Mistral capacity, and prepending domain names to training data significantly boosts how much the model retains, hinting at implicit domain prioritization.
Implications for scaling: If the 2-bit ceiling is real, compute-efficient scaling should match parameter count to the amount of factual knowledge you want to store, and quantization can compress weights without losing knowledge.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack