YaRN (Efficient Context Extension)

Rotary-embedding scaling: Extends RoPE-based context length via a combined attention and NTK-aware scaling scheme, avoiding the degradation of naive interpolation.

Fine-tune extrapolation: Extrapolates meaningfully beyond the limited context seen during fine-tuning, so short fine-tune sequences can unlock much longer inference contexts.

128K context: Successfully scales Llama-family models to 128K-token context with minimal additional training compute.

Open recipe: Adopted widely across the open-source community as a standard recipe for extending Llama and other RoPE-based LLMs.