Language Models Need Sleep
Free while signed in. Answers cite the passages they came from.
Attention scales badly with context length, so long-horizon agents keep paying a growing cost as their context grows. This paper studies a sleep-like consolidation mechanism: the model periodically converts recent context into persistent fast weights, then clears its key-value cache. During the sleep phase it performs offline recurrent passes over the accumulated context and updates fast weights in its state-space blocks through a learned local rule.
Consolidate, then clear the cache: Recent context is folded into fast weights stored in the model's SSM blocks before the KV cache is discarded. The agent keeps what it learned without carrying the full attention bill into every future step.
Compute moves to sleep, latency stays at wake: The extra work happens offline during consolidation, so wake-time prediction keeps its low latency. The tradeoff is explicit and controllable rather than hidden in a ballooning context window.
More sleep helps the hardest cases: Increasing sleep duration improves performance, with the largest gains precisely on tasks that require the most complex reasoning over long histories. The mechanism buys the most where naive attention struggles most.
Why it matters: Long-horizon agents are the first systems to feel the quadratic cost of context. A biologically inspired consolidation step gives a principled alternative to ever-longer context windows, and it maps cleanly onto the state-space architectures already used for efficiency.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack