Training LLMs over Neurally Compressed Text
Free while signed in. Answers cite the passages they came from.
The paper proposes Equal-Info Windows, a neural compression scheme that segments text into equal-bit-length blocks so an LLM can train directly on compressed bytes without losing learnability.
Equal-Info Windows: Text is split into windows that each compress to the same number of bits, turning arithmetic-coded output into a stable sequence that a transformer can learn from.
Why naive compression fails: Standard arithmetic coding produces sequences whose boundaries shift with context, making training unstable; Equal-Info Windows restores the positional regularity LLMs rely on.
Beats byte-level, trails BPE: At scale the method outperforms byte-level baselines by a wide margin on perplexity and inference speed, but still trails traditional subword tokenizers at matched parameter counts.
Shorter sequences, faster inference: Because each token encodes more raw text, autoregressive generation produces the same output in fewer steps, cutting latency meaningfully.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack