ConvNets Match Vision Transformers
First page
Paper summary
DeepMind shows that strong ConvNet architectures pretrained at scale match ViTs on ImageNet performance at comparable compute.
Ask this paper
01
JFT-4B pretraining: Pretrains performant ConvNet architectures (NFNets) on JFT-4B at scale - matching the data regime where ViTs typically pull ahead.
02
Log-log scaling law: Observes a log-log scaling law between held-out loss and compute, mirroring the scaling properties seen in ViTs.
03
ImageNet parity: Fine-tuned NFNets match the reported performance of Vision Transformers at comparable compute budgets, refuting the "ConvNets don't scale" narrative.
04
Architecture vs. recipe: Argues that the ConvNet-vs-ViT gap is largely a scale/recipe gap rather than an architectural limitation - a recurring theme in vision research.