Transformers as Support Vector Machines

Hard-margin SVM connection: Shows the optimization geometry of self-attention in transformers exhibits a direct connection to hard-margin SVM problems.

Implicit regularization: Gradient descent without early stopping leads to implicit regularization, with attention converging toward SVM-like solutions.

Theoretical foundation: Provides a rare closed-form theoretical lens on self-attention dynamics, cutting through much of the "transformers as black box" framing.

Future analysis tool: The SVM connection gives researchers a principled tool to analyze attention convergence, generalization, and feature selection.