RA-DIT (Retrieval-Augmented Dual Instruction Tuning)
Free while signed in. Answers cite the passages they came from.
Meta's RA-DIT is a lightweight recipe that retrofits LLMs with retrieval capabilities through dual fine-tuning.
Two-stage fine-tuning: Stage 1 updates the LM to better use retrieved information; stage 2 updates the retriever to return documents the LM actually prefers.
Each stage adds gains: Both stages contribute meaningfully and combine to produce strong downstream RAG performance without end-to-end joint training.
65B SoTA: The 65B model achieves state-of-the-art on a range of knowledge-intensive zero-shot and few-shot benchmarks.
Strong relative gains: Outperforms existing retrieval-augmented approaches by up to +8.9% in zero-shot and +1.4% in 5-shot settings - non-trivial gains on already-strong baselines.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack