HOLA
Free while signed in. Answers cite the passages they came from.
Linear-attention and state-space models compress an entire prefix into a fixed-size state, buying constant memory but overwriting earlier facts when many key-value associations compete. HOLA gives linear attention a hippocampal complement, pairing a compressive recurrent state with a small exact memory to recover long-range recall.
Two memories, different jobs: HOLA keeps the usual delta-rule state as compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory where each store handles what it is best at.
Selective, learning-free writes: The cache writes without a learned eviction module, keeping only tokens whose prediction residual was actually committed to the state, so it stores exactly the associations that should not be forced through compression.
Strong recall at small scale: At 340M parameters on 15B SlimPajama tokens, it lowers Wikitext perplexity from 27.32 to 22.92, below a full-attention Transformer++ at 26.88, and stays robust on RULER needle recall out to 32k tokens, 16 times its training length.
Why it matters: It shows you can keep linear attention's efficiency and still recover the exact recall that pure compression destroys, a practical path for long-context models that cannot afford full attention.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack