Research map / Large language models / Retrieval and knowledge
Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoRefine, an RL post-training framework where the LLM interleaves search calls with explicit knowledge-refinement steps and is trained with GRPO using answer F1 and a retrieval-specific reward on refine blocks.
Problem
Retrieval-augmented reasoning with LLMs often retrieves noisy or irrelevant documents that distract multi-hop reasoning, and existing RL methods rely only on outcome rewards, giving no direct supervision of retrieval quality.
Approach
Adds an explicit refine step between search and answer within the reasoning loop ("search-and-refine-during-think") and rewards the refine content (whether ground-truth answer appears in refinements) alongside answer F1, with retrieved documents masked from the loss under GRPO.
Key results
- AutoRefine-Base (Qwen2.5-3B) reaches 0.405 average EM across seven QA benchmarks vs. 0.312 for Search-R1-Base and 0.319 for ReSearch-Base.
- AutoRefine-Instruct averages 0.396 EM vs. 0.336 for the strongest baseline Search-R1-Instruct, a 0.060 gain.
- Removing the retrieval reward drops AutoRefine-Base average EM from 0.405 to 0.376; also removing refinement drops it to 0.312.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoRefine instead of the strongest baseline | Qwen2.5-3B QA retrieval reasoning · Seven QA benchmarks (Avg.) | accuracy: 0.336 -> 0.405 | Helped |
| Use AutoRefine instead of the strongest baseline | Qwen2.5-3B QA retrieval reasoning · 2Wiki | accuracy: 0.310 -> 0.393 | Helped |
| Add retrieval-specific reward (ablation) | AutoRefine-Base · Seven QA benchmarks (Avg.) | accuracy: 0.376 -> 0.405 | Helped |
| Add knowledge refinement step alongside retrieval-specific reward (ablation) | AutoRefine-Base · Seven QA benchmarks (Avg.) | accuracy: 0.312 -> 0.405 | Helped |
| Add retrieval-specific reward (ablation) | AutoRefine-Instruct · Seven QA benchmarks (Avg.) | accuracy: 0.360 -> 0.396 | Helped |
| Add knowledge refinement step alongside retrieval-specific reward (ablation) | AutoRefine-Instruct · Seven QA benchmarks (Avg.) | accuracy: 0.333 -> 0.396 | Helped |
| Use nonlinear rather than linear answer/retrieval reward combination (ablation) | AutoRefine with reward on refinement steps · Seven QA benchmarks (Avg.) | accuracy: 0.375 -> 0.405 | Helped |
| Apply retrieval reward to refinement rather than retrieved documents (ablation) | AutoRefine with nonlinear reward combination · Seven QA benchmarks (Avg.) | accuracy: 0.384 -> 0.405 | Helped |
| Add an external BART refiner (ablation) | Search-R1 · PopQA | accuracy: 0.413 -> 0.450 | Helped |
| Add an external BART refiner (ablation) | Search-R1 · 2Wiki | accuracy: 0.274 -> 0.239 | Hurt |
| Replace CEM retrieval reward with word-level recall (ablation) | AutoRefine on complex answers (>5 words) · Complex answers (Avg.) | accuracy: 0.177 -> 0.216 | Helped |
| Increase base model size from Qwen2.5-3B to Qwen2.5-7B (ablation) | AutoRefine · Seven QA benchmarks (Avg.) | accuracy: 0.405 -> 0.455 | Helped |
Surprising findings
- Directly rewarding retrieved documents gives only marginal gains over answer-only reward, while rewarding refinement steps gives the largest improvement.
- A linear combination of answer and refinement rewards performs worse than the non-linear reward that gives partial credit only when the answer is wrong.
Limitations
- Evaluation relies only on exact match and F1, which may miss semantically correct answers with textual variation.
- Retrieval uses a fixed December 2018 Wikipedia snapshot, so it lacks time-sensitive information.
Experimental setup
Datasets: NQ HotpotQA TriviaQA PopQA 2WikiMultihopQA Musique Bamboogle
Benchmarks: NQ TriviaQA PopQA HotpotQA 2WikiMultihopQA Musique Bamboogle
Metrics: exact match F1 cover exact match search success rate
Baselines: Direct Generation SFT R1-Instruct R1-Base Naive RAG Search-o1 IRCoT ReSearch Search-R1
Builds on: Search-R1 GRPO ReSearch Search-o1 IRCoT
Closest papers by meaning
- ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards (ICML 2026 · similarity 0.89)
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration (ICML 2026 · similarity 0.88)
- Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging (NeurIPS 2025 · similarity 0.88)
- Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs (ICLR 2026 · similarity 0.88)
- Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.