StreamingLLM

Attention sink phenomenon: The authors observe that attention heads consistently route a large fraction of attention mass to the first few tokens, even when those tokens are semantically irrelevant.

Sink tokens are essential: Keeping the KV states of initial tokens around dramatically recovers the performance of sliding-window attention.

Infinite-length inference: Enables LLMs trained with finite context to generate infinitely long outputs without fine-tuning, by retaining sink tokens plus a sliding window.

Emergent explanation: Attention sinks appear because the softmax must normalize to one - unused attention mass is "dumped" onto the first tokens, which explains why removing them breaks the model.