Retentive Network (RetNet)

Three-mode formulation: Supports parallel training, recurrent inference, and chunkwise recurrent representation - combining Transformer-style training with RNN-style inference.

O(1) inference cost: Achieves constant-memory inference per step via the recurrent form, dramatically cheaper than attention's O(n) per-token cost.

Retention mechanism: Replaces softmax attention with an exponentially-decaying retention kernel that supports both parallel and recurrent computation.

Post-Transformer contender: Positioned alongside Mamba, RWKV, and Hyena as one of the credible attempts to dethrone attention - though attention remained dominant through 2024.