Research map / Large language models / Reasoning and chain of thought
ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains LLMs to interleave text reasoning with Wikipedia search calls using GRPO and rule-based F1 rewards, without any supervised reasoning data, trained only on MuSiQue.
Problem
Multi-hop question answering requires interleaving reasoning with multiple retrieval steps, but existing multi-step RAG methods rely on hand-designed prompts or heuristics, and labeling reasoning-with-search trajectories is costly.
Approach
Treat search queries and retrieved results as part of the reasoning chain, use GRPO with outcome-only F1 rewards so the policy learns when and how to search from scratch, and mask retrieval tokens from the loss.
Key results
- ReSearch-Qwen-7B-Instruct reaches 43.52 EM on HotpotQA and 47.59 EM on 2Wiki, versus best baseline Iter-RetGen at 34.36 and 27.92 EM (7B setting).
- Average improvement over the best baseline is 15.81% EM and 17.56% LJ for the 7B model, and 14.82% EM and 15.46% LJ for the 32B model.
- Trained only on MuSiQue, ReSearch-Qwen-32B-Instruct reaches 46.73 EM on HotpotQA and 56.80 EM on Bamboogle, showing cross-dataset generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using Iter-RetGen | Qwen2.5-32B-Instruct · HotpotQA dev | EM (%): 39.81 -> 46.73 | Helped |
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using Iter-RetGen | Qwen2.5-32B-Instruct · HotpotQA dev | LJ (%): 58.80 -> 67.70 | Helped |
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using Iter-RetGen | Qwen2.5-32B-Instruct · 2WikiMultiHopQA dev | EM (%): 33.64 -> 44.90 | Helped |
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using Iter-RetGen | Qwen2.5-32B-Instruct · 2WikiMultiHopQA dev | LJ (%): 38.22 -> 50.30 | Helped |
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using Iter-RetGen | Qwen2.5-32B-Instruct · MuSiQue dev | EM (%): 12.49 -> 26.40 | Helped |
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using Iter-RetGen | Qwen2.5-32B-Instruct · MuSiQue dev | LJ (%): 20.11 -> 38.56 | Helped |
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using IRCoT | Qwen2.5-32B-Instruct · Bamboogle test | EM (%): 31.20 -> 56.80 | Helped |
| Train Qwen2.5-32B-Instruct with ReSearch reinforcement learning instead of using IRCoT | Qwen2.5-32B-Instruct · Bamboogle test | LJ (%): 47.20 -> 67.20 | Helped |
| Train ReSearch from an instruction-tuned model rather than its base model | ReSearch Qwen2.5-7B · HotpotQA dev | EM (%): 40.57 -> 43.52 | Helped |
Surprising findings
- Reflection and self-correction (e.g., 'I made a mistake' then rewriting the search query) emerge during RL without being prompted or trained for.
- IRCoT with Qwen2.5-32B-Instruct scores far lower than naive RAG on 2Wiki EM (13.53 vs 30.38), suggesting prompt-based multi-step RAG is fragile.
Limitations
- Focuses on short answers verifiable by F1, not longer open-ended responses.
- Uses only Wikipedia (Dec 2018) as the knowledge base; other domain-specific corpora are untested.
- Evaluation of LJ relies on gpt-4o-mini as judge; the small Bamboogle test set (125 samples) makes differences noisy.
Experimental setup
Datasets: MuSiQue Wikipedia Dec. 2018 (FlashRAG index)
Benchmarks: HotpotQA 2WikiMultiHopQA MuSiQue Bamboogle
Metrics: exact match LLM-as-a-judge accuracy
Baselines: Naive Generation Naive RAG Iter-RetGen IRCoT
Builds on: GRPO DeepSeek-R1 verl FlashRAG IRCoT
Closest papers by meaning
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.91)
- Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.91)
- Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.91)
- Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion (ICLR 2026 · similarity 0.90)
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play (NeurIPS 2025 · similarity 0.90)
- QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.