Retentive Network (RetNet)
Free while signed in. Answers cite the passages they came from.
Microsoft's proposed foundation architecture aiming to replace Transformer attention for LLMs.
Three-mode formulation: Supports parallel training, recurrent inference, and chunkwise recurrent representation - combining Transformer-style training with RNN-style inference.
O(1) inference cost: Achieves constant-memory inference per step via the recurrent form, dramatically cheaper than attention's O(n) per-token cost.
Retention mechanism: Replaces softmax attention with an exponentially-decaying retention kernel that supports both parallel and recurrent computation.
Post-Transformer contender: Positioned alongside Mamba, RWKV, and Hyena as one of the credible attempts to dethrone attention - though attention remained dominant through 2024.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack