End-to-End Test-Time Training for Long Context
Free while signed in. Answers cite the passages they came from.
This paper reframes long-context language modeling as a continual learning problem rather than architecture design. TTT-E2E uses a standard Transformer with sliding-window attention that continues learning at test time via next-token prediction, compressing context into its weights rather than storing all key-value pairs.
Test-time training approach: The model learns at test time by predicting next tokens on the given context, compressing information into weights. This is combined with meta-learning at training time to prepare the model’s initialization for test-time learning.
End-to-end in two ways: The inner loop directly optimizes next-token prediction loss, while the outer loop optimizes the final loss after TTT via gradients of gradients. This contrasts with prior TTT methods and dynamic evaluation approaches.
Scaling with context length: For 3B models trained on 164B tokens, TTT-E2E scales with context length the same way as full attention Transformers, while alternatives like Mamba 2 and Gated DeltaNet do not maintain performance in longer contexts.
Efficient inference: Similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7x faster than full attention for 128K context on H100 while achieving comparable or better loss.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack