FlashAttention-2
Free while signed in. Answers cite the passages they came from.
Tri Dao's follow-up to FlashAttention, dramatically improving attention throughput on modern GPUs.
Work partitioning: Redesigns parallelism so non-matmul FLOPs are reduced and thread blocks are better utilized across SMs.
~2x speedup: Achieves approximately 2x speedup over FlashAttention-1 and reaches 50-73% of theoretical maximum FLOPs/s on A100.
Shared-memory communication: Parallelizes attention along sequence length, increases occupancy, and reduces cross-warp communication via shared memory.
Training infrastructure staple: Became the default attention kernel in PyTorch, HuggingFace, vLLM, and nearly every 2024 training stack for long-context models.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack