Research map / Large language models / Instruction and fine-tuning
AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a single LLM as decomposer and solver for search-augmented reasoning via SFT on 180K mixed examples followed by iterative DPO-based self-play using final-answer exact-match rewards.
Problem
Search-augmented LLMs struggle with complex multi-hop reasoning due to ineffective multi-hop retrieval and limited reasoning ability, and existing RL search methods are memory-intensive and rely on QA-only supervision.
Approach
A cooperative self-play framework where one LLM alternates between a decomposer (produces subquestion templates) and a solver (answers subquestions with retrieved context), trained with SFT then iterative DPO using preference pairs built from final-answer exact-match rewards, without intermediate annotations.
Key results
- AceSearcher-32B achieves the highest overall QA+fact-verification average (60.7), reported as outperforming baselines by up to 7.6%.
- On DocMath-Eval, AceSearcher-32B reaches 66.1 average accuracy versus DeepSeek-V3's 66.4 with under 5% of its parameters.
- AceSearcher-8B reaches 59.6 average overall, and the 1.5B variant reaches 49.4, which the paper says matches or exceeds 8B baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reinforcement fine-tuning (RFT) (ablation) | AceSearcher with Llama-3.1-8B-Instruct · QA and fact verification | Avg. EM: 56.2 -> 59.6 | Helped |
| add reinforcement fine-tuning (RFT) (ablation) | AceSearcher with Llama-3.1-8B-Instruct · DocMath-Eval | Avg. Accuracy: 52.3 -> 59.0 | Helped |
| add supervised fine-tuning (SFT) (ablation) | AceSearcher with Llama-3.1-8B-Instruct · QA and fact verification | Avg. EM: 47.6 -> 59.6 | Helped |
| add supervised fine-tuning (SFT) (ablation) | AceSearcher with Llama-3.1-8B-Instruct · DocMath-Eval | Avg. Accuracy: 48.0 -> 59.0 | Helped |
| add the trained decomposer instead of replacing it with Llama-3.1-8B-Instruct (ablation) | AceSearcher with Llama-3.1-8B-Instruct · QA and fact verification | Avg. EM: 55.3 -> 59.6 | Helped |
| add the trained solver instead of replacing it with Llama-3.1-8B-Instruct (ablation) | AceSearcher with Llama-3.1-8B-Instruct · QA and fact verification | Avg. EM: 40.9 -> 59.6 | Helped |
| include search-related data in SFT (ablation) | AceSearcher with Llama-3.1-8B-Instruct · QA and fact verification | Avg. EM: 49.3 -> 59.6 | Helped |
| include reasoning data in SFT (ablation) | AceSearcher with Llama-3.1-8B-Instruct · DocMath-Eval | Avg. Accuracy: 53.5 -> 59.0 | Helped |
| use the full diverse SFT mixture instead of context-aware QA tasks alone (ablation) | AceSearcher with Llama-3.1-8B-Instruct · QA and fact verification | Avg. EM: 36.3 -> 59.6 | Helped |
| use iterative preference optimization with positive and negative trajectories (ablation) | AceSearcher with Llama-3.1-8B-Instruct · QA and fact verification | Avg. EM: 56.5 -> 59.6 | Helped |
| use iterative preference optimization with positive and negative trajectories (ablation) | AceSearcher with Llama-3.1-8B-Instruct · DocMath-Eval | Avg. Accuracy: 56.6 -> 59.0 | Helped |
| increase the number of RFT iterations to the third iteration (ablation) | AceSearcher 8B · RFT | performance | No clear effect |
Surprising findings
- Reasoning models such as Qwen-3 and DeepSeek-R1-Distill give limited gains on multi-hop retrieval, suggesting long thinking does not solve multi-hop retrieval.
- Online iterative DPO outperforms offline DPO, RAFT, REST-EM and SimPO in the RL ablation.
Limitations
- Retriever is fixed during training and inference; joint retrieval-reasoning optimization is left to future work.
- Decomposition pipeline adds inference overhead that may hurt latency-sensitive use.
- Uses iterative DPO as a practical substitute for fully online RL due to resource constraints.
Experimental setup
Datasets: NQ SQuAD DROP NarrativeQA Quoref ROPES FEVER TAT-QA GSM8K ConvFinQA StrategyQA TabMWP
Benchmarks: 2WikiMHQA HotpotQA Bamboogle MusiQue HOVER ExFever DocMath-Eval
Metrics: exact match accuracy F1 human evaluation score
Baselines: Llama-3.1-8B-Instruct RAG DeepSeek-R1-Distill RAG Qwen-3 RAG Plan-RAG Search-R1 R1-Searcher DeepResearcher InstructRAG CORAG IRCOT Search-o1 IterDRAG
Builds on: DPO Search-R1 IterDRAG InstructRAG LoRA
Closest papers by meaning
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Textual Space (ICLR 2026 · similarity 0.88)
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.88)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.88)
- Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers (NeurIPS 2025 · similarity 0.87)
- Accelerating Large Language Model Reasoning via Speculative Search (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.