Efficient Attention Mechanisms

This survey reviews linear and sparse attention techniques that reduce the quadratic cost of Transformer self-attention, enabling more efficient long-context modeling. It also examines their integration into large-scale LLMs and discusses practical deployment and hardware considerations.