Giraffe
Free while signed in. Answers cite the passages they came from.
A family of context-extended Llama and Llama 2 models, along with an empirical study of context-extension techniques.
Extended contexts: Fine-tuned models with 4K, 16K, and 32K context windows, providing ready-to-use open long-context variants.
Technique comparison: Systematically compares context-extension methods including positional interpolation, truncation strategies, and attention scaling.
Practitioner insights: Reports practical findings on which techniques preserve downstream quality at extended contexts - useful for anyone building long-context applications.
Context-extension recipe: The lessons from Giraffe fed directly into the recipes that would culminate in YaRN and similar approaches later that year.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack