Continuous Memory Machines
Ciaran Regan, Kai Arulkumaran, Luke Darlow, Stefania Druga, Sebastian Risi and Llion Jones at Sakana AI introduce the Continuous Memory Machine (CMM), a recurrent architecture with separate matrix-valued short-term and long-term memory states, built on Sakana's Continuous Thought Machine. It appears at the NeurIPS 2026 PALM workshop.
Ask this paper
Problem. Standard RNNs store the whole history in one hidden vector, so short-term computation and long-term retention compete for the same representation.
Two memories. The short-term memory tracks recent neural activity that neuron-level models use for computation; a persistent long-term memory stores information for later steps.
Read-write mechanism. A Transformer updates both memory stores jointly, so each store can reorganize itself and read from and write to the other.
Results. Across algorithmic tasks (copy, associative recall, priority sort), few-shot regression and maze solving, the CMM beats a suite of baselines including LSTM, DNC, RMC and CTM, and generalizes to longer inputs better than earlier memory-augmented networks.
Ablations and cost. Replacing the CTM short-term processing with an LSTM lowers performance throughout. Attention analysis shows the model uses long-term memory for algorithmic and in-context tasks and skips it when not needed; the joint memory update adds compute and memory cost.
Abstract
Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of rapid neuron-level processing and longer-term retention found in biology. To that end, we introduce the Continuous Memory Machine (CMM), a recurrent architecture with matrix-valued short- and long-term memory states serving distinct functional roles. Building on the Continuous Thought Machine (CTM), the CMM's short-term memory tracks recent neural activity, with uniquely parameterized neuron-level models learning to use these activity patterns for computation. A persistent long-term memory stores information for later use, with a Transformer jointly updating both memory stores, providing an expressive bidirectional read--write mechanism such that each store can reorganize its own contents and both read from and write to the other. Across algorithmic, in-context learning, and recurrent reasoning tasks, the CMM outperforms a broad suite of baselines, exhibiting stronger generalization than prior memory-augmented networks while preserving the CTM's interpretable attention patterns. Code is available at https://github.com/SakanaAI/continuous-memory-machines.