Phase Transition in Dot-Product Attention
Free while signed in. Answers cite the passages they came from.
A theoretical paper that analyzes a solvable low-rank tied-QK attention model and uncovers a data-driven phase transition between positional and semantic attention regimes.
Solvable toy model: Low-rank, tied-query-and-key dot-product attention that admits a closed-form characterization of the loss landscape's global minimum.
Two attention regimes: Identifies positional attention (attention depends only on token positions) and semantic attention (attention uses token content), both of which can be optimal depending on data.
Data-threshold transition: Below a critical data budget, the network falls back to positional attention; above it, it transitions to semantic attention, producing an abrupt phase-transition behavior.
Beats linear baselines: With enough data, the dot-product attention layer operating in the semantic regime outperforms any linear positional baseline, clarifying *why* attention is powerful.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack