Retrieval Meets Long-Context LLMs
Free while signed in. Answers cite the passages they came from.
NVIDIA's study comparing RAG and long-context LLMs, with the punchline that the two are complementary rather than substitutes.
4K + RAG ≈ 16K fine-tuned: An LLM with only a 4K context window using simple RAG can match a fine-tuned LLM with 16K context - a striking efficiency result.
Retrieval always helps: Retrieval improves performance regardless of context-window size, even when the model can fit the full document in its native context.
LLaMA-2 70B beats GPT-3.5: A retrieval-augmented LLaMA 2 70B with 32K context outperforms GPT-3.5-turbo-16k on seven long-context tasks including QA and query-based summarization.
Implication: Don't think of long context and retrieval as competing solutions - pair them, and let the model attend to both the query and retrieved evidence.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack