Language Modeling Is Compression
Free while signed in. Answers cite the passages they came from.
DeepMind empirically revisits the theoretical equivalence between prediction and compression, applied to modern LLMs.
Theoretical equivalence: Reminds that optimal compression and optimal prediction are duals - a good language model is implicitly a powerful compressor.
ImageNet compression: Chinchilla 70B compresses ImageNet patches to 43.4% of raw size, better than domain-specific codecs like PNG.
LibriSpeech compression: Compresses LibriSpeech samples to 16.4% of raw size, beating FLAC and gzip on audio data despite never being trained on audio.
Cross-modal generalization: Shows LLMs work as general-purpose compressors across text, image, and audio - a striking demonstration of in-context learning's reach.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack