ConvNets Match Vision Transformers
Free while signed in. Answers cite the passages they came from.
DeepMind shows that strong ConvNet architectures pretrained at scale match ViTs on ImageNet performance at comparable compute.
JFT-4B pretraining: Pretrains performant ConvNet architectures (NFNets) on JFT-4B at scale - matching the data regime where ViTs typically pull ahead.
Log-log scaling law: Observes a log-log scaling law between held-out loss and compute, mirroring the scaling properties seen in ViTs.
ImageNet parity: Fine-tuned NFNets match the reported performance of Vision Transformers at comparable compute budgets, refuting the "ConvNets don't scale" narrative.
Architecture vs. recipe: Argues that the ConvNet-vs-ViT gap is largely a scale/recipe gap rather than an architectural limitation - a recurring theme in vision research.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack