Search-R1
Free while signed in. Answers cite the passages they came from.
This paper tackles search-augmented reasoning by teaching LLMs to query a search engine multiple times—while they reason—using reinforcement learning. Key ideas include:
Multi-turn retrieval – The LLM can interleave text generation with repeated calls to a search engine, refining queries at each step. This differs from simple one-shot retrieval-augmented generation (RAG).
Fully RL-based training – Unlike prior “tool-using” approaches that need large supervised datasets, the authors rely on outcome rewards only. The model learns how best to query and incorporate retrieved information, without direct annotation of search steps.
Retrieved token masking – To stabilize training, the authors ensure only model-generated tokens are optimized in the policy gradient, preventing the search engine’s returned text from skewing the RL updates.
Impressive gains – Across seven QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, etc.), Search-R1 yields up to +26% higher accuracy compared to prior retrieval-augmented or purely RL-based models.
Flexible across architectures – The framework works on both “base” and “instruct” variants of Qwen and LLaMA, showing its general applicability. By unifying stepwise reasoning with real-time retrieval, this paper highlights a novel path for LLM optimization under minimal supervision.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack