Scaling Transformer to 1 Billion Tokens (LongNet)

Dilated attention: Introduces dilated attention that exponentially grows the attention field, enabling linear complexity in sequence length.

No short-sequence loss: Achieves extreme long-context scaling with no degradation on shorter sequences.

1B token demo: Demonstrates viability at the 1-billion token context scale - an order of magnitude beyond anything previously attempted.

Long-context frontier: Pushed the frontier of what's theoretically possible for ultra-long-context Transformers, even though production models stayed in the hundreds-of-thousands-of-tokens range.