Phase Transition in Dot-Product Attention
A theoretical paper that analyzes a solvable low-rank tied-QK attention model and uncovers a data-driven phase transition between positional and semantic attention regimes.
Ask this paper
Solvable toy model: Low-rank, tied-query-and-key dot-product attention that admits a closed-form characterization of the loss landscape's global minimum.
Two attention regimes: Identifies positional attention (attention depends only on token positions) and semantic attention (attention uses token content), both of which can be optimal depending on data.
Data-threshold transition: Below a critical data budget, the network falls back to positional attention; above it, it transitions to semantic attention, producing an abrupt phase-transition behavior.
Beats linear baselines: With enough data, the dot-product attention layer operating in the semantic regime outperforms any linear positional baseline, clarifying *why* attention is powerful.