Training LLMs over Neurally Compressed Text
The paper proposes Equal-Info Windows, a neural compression scheme that segments text into equal-bit-length blocks so an LLM can train directly on compressed bytes without losing learnability.
Ask this paper
Equal-Info Windows: Text is split into windows that each compress to the same number of bits, turning arithmetic-coded output into a stable sequence that a transformer can learn from.
Why naive compression fails: Standard arithmetic coding produces sequences whose boundaries shift with context, making training unstable; Equal-Info Windows restores the positional regularity LLMs rely on.
Beats byte-level, trails BPE: At scale the method outperforms byte-level baselines by a wide margin on perplexity and inference speed, but still trails traditional subword tokenizers at matched parameter counts.
Shorter sequences, faster inference: Because each token encodes more raw text, autoregressive generation produces the same output in fewer steps, cutting latency meaningfully.