δ-mem
δ-mem augments a frozen full-attention model with a compact online associative-memory state. The state is a fixed-size matrix updated by delta-rule learning during generation, and its readout produces low-rank corrections to the backbone's attention output. There is no fine-tuning, no backbone swap, and no context extension.
Ask this paper
Frozen backbone: The base model weights are unchanged. δ-mem adds a small online state plus a pair of low-rank read and write projections.
Delta-rule update integrated into attention: The memory matrix is updated by delta-rule learning during generation, and the readout produces additive query and output corrections to the attention computation rather than functioning as a separate retrieval step.
Results from an 8x8 state: An 8x8 online memory lifts the frozen backbone's average score by 1.10x and beats the strongest non-δ-mem memory baseline by 1.15x. On memory-heavy benchmarks the gap widens: 1.31x on MemoryAgentBench and 1.20x on LoCoMo. General capabilities are largely preserved.
Why it matters: The mechanism offers an alternative to context extension and external retrieval for long-horizon memory, with minimal deployment overhead on frozen frontier models.