The Spike, the Sparse and the Sink
Yann LeCun and collaborators at NYU dissect two recurring phenomena in Transformer language models: massive activations, where a small number of tokens exhibit extreme outliers in specific channels, and attention sinks, where certain tokens attract disproportionate attention mass regardless of semantic relevance. The paper reveals that their co-occurrence is largely an architectural artifact.
Ask this paper
Distinct operational scopes: Massive activations operate globally, inducing near-constant hidden representations that persist across layers and function as implicit model parameters. Attention sinks operate locally, modulating attention outputs across heads and biasing individual heads toward short-range dependencies.
Pre-norm as the critical factor: The pre-norm configuration common in modern Transformers is identified as the key architectural element enabling the co-occurrence of these two phenomena. Removing pre-norm causes massive activations and attention sinks to decouple entirely.
Practical implications for efficiency: Understanding these phenomena has direct consequences for model compression, quantization, and KV-cache optimization. Many efficiency techniques fail silently when they inadvertently disrupt massive activations or attention sinks, and this paper explains why.
Not functionally necessary: The co-occurrence of spikes and sinks is a design-dependent artifact rather than a fundamental requirement for model performance. This opens the door to architectural modifications that could eliminate these phenomena without sacrificing capability.