Scaling MLPs: A Tale of Inductive Bias

Pure-MLP scaling: Demonstrates that large pure-MLP models trained on enough data can reach surprisingly strong performance on image classification.

Inductive bias is compensable: Challenges the dogma that CNN/Transformer inductive biases are necessary - scale and data can substitute.

Bitter lesson evidence: Adds to the "bitter lesson" empirical evidence that general methods leveraging computation outperform those leveraging human-designed priors.

Architecture agnosticism: Part of the 2023 trend showing that many architectures (MLPs, State Space Models, RNNs, Transformers) converge at scale.