Scaling Transformer to 1M tokens with RMT

Recurrent memory mechanism: Augments BERT with a recurrent memory that carries information across segments, enabling massive context lengths.

2M token context: Scales effective context to two million tokens while maintaining high memory retrieval accuracy.

Segment-level recurrence: Processes input in segments while passing a compressed memory token stream across them.

Long-context trend: Part of the 2023 explosion of long-context techniques that established ultra-long context as a viable research direction.