Leave No Context Behind (Infini-attention)
Free while signed in. Answers cite the passages they came from.
Google's Infini-attention extends Transformer LLMs to effectively infinite context with bounded memory and compute. It blends a compressive memory module with both masked local attention and linear long-term attention inside a single Transformer block.
Unbounded context, bounded memory: The compressive memory stores historical key-value statistics in a fixed-size matrix, so token streaming does not grow memory over time and enables true long-form inference.
Dual-attention block: Each Infini-attention layer combines local windowed attention for recent tokens with long-term linear attention over the compressive memory, giving the same block access to short- and long-range dependencies simultaneously.
Empirical results: 1B and 8B Infini-Transformers outperform baseline long-context models on book summarization (500K tokens) and 1M-token passkey retrieval, while achieving a 114x memory compression ratio on long-context language modeling.
Streaming implication: Because the memory footprint is bounded, Infini-Transformer is a clean fit for streaming inference, document-flow agents, and any setting where the input grows indefinitely.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack