Hierarchical Vision Transformer (Hiera)
Free while signed in. Answers cite the passages they came from.
First page
The curator’s take
Key pointsPretrains ViTs with MAE while removing unnecessary multi-stage complexity.
01
Simplified architecture: Strips away hand-designed components (shifted windows, relative position biases) from hierarchical ViTs like Swin.
02
MAE pretraining: Leverages masked autoencoder pretraining to compensate for reduced inductive bias.
03
Faster and more accurate: Achieves better accuracy and faster inference/training than prior hierarchical ViTs.
04
Architecture minimalism: Reinforces the "bitter lesson" direction - simpler architectures with better pretraining beat complex hand-designed ones.
Every Monday
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack