Retrieval Meets Long-Context LLMs
NVIDIA's study comparing RAG and long-context LLMs, with the punchline that the two are complementary rather than substitutes.
Ask this paper
4K + RAG ≈ 16K fine-tuned: An LLM with only a 4K context window using simple RAG can match a fine-tuned LLM with 16K context - a striking efficiency result.
Retrieval always helps: Retrieval improves performance regardless of context-window size, even when the model can fit the full document in its native context.
LLaMA-2 70B beats GPT-3.5: A retrieval-augmented LLaMA 2 70B with 32K context outperforms GPT-3.5-turbo-16k on seven long-context tasks including QA and query-based summarization.
Implication: Don't think of long context and retrieval as competing solutions - pair them, and let the model attend to both the query and retrieved evidence.