Architecture · Multimodal

Hierarchical Vision Transformer (Hiera)

First page

Paper summary

Pretrains ViTs with MAE while removing unnecessary multi-stage complexity.

Ask this paper

Key points

Simplified architecture: Strips away hand-designed components (shifted windows, relative position biases) from hierarchical ViTs like Swin.

MAE pretraining: Leverages masked autoencoder pretraining to compensate for reduced inductive bias.

Faster and more accurate: Achieves better accuracy and faster inference/training than prior hierarchical ViTs.

Architecture minimalism: Reinforces the "bitter lesson" direction - simpler architectures with better pretraining beat complex hand-designed ones.

Every Monday