No Positional Encodings (NoPE)
First page
Paper summary
Shows explicit position embeddings aren't essential for decoder-only Transformers.
Ask this paper
01
Implicit positional learning: Decoder-only Transformers learn positional information from the causal attention mask alone - no explicit encoding needed.
02
Length generalization: NoPE generalizes better to longer sequences than ALiBi and Rotary, which have surprising length-generalization issues.
03
Architectural simplification: Removing positional encodings simplifies the architecture with no quality loss on standard tasks.
04
Long-context influence: Informed the 2024 resurgence of interest in length-generalization-friendly architectures.