Search-R1
This paper tackles search-augmented reasoning by teaching LLMs to query a search engine multiple times—while they reason—using reinforcement learning. Key ideas include:
Ask this paper
Multi-turn retrieval – The LLM can interleave text generation with repeated calls to a search engine, refining queries at each step. This differs from simple one-shot retrieval-augmented generation (RAG).
Fully RL-based training – Unlike prior “tool-using” approaches that need large supervised datasets, the authors rely on outcome rewards only. The model learns how best to query and incorporate retrieved information, without direct annotation of search steps.
Retrieved token masking – To stabilize training, the authors ensure only model-generated tokens are optimized in the policy gradient, preventing the search engine’s returned text from skewing the RL updates.
Impressive gains – Across seven QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, etc.), Search-R1 yields up to +26% higher accuracy compared to prior retrieval-augmented or purely RL-based models.
Flexible across architectures – The framework works on both “base” and “instruct” variants of Qwen and LLaMA, showing its general applicability. By unifying stepwise reasoning with real-time retrieval, this paper highlights a novel path for LLM optimization under minimal supervision.