Research map / Reinforcement learning / Visual and representation RL
SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmartRAG, a policy-network-plus-retriever RAG pipeline that decides when to retrieve, rewrites queries, and answers, jointly optimized with PPO using environment feedback and a retrieval penalty.
Problem
RAG systems have multiple modules (retriever, generator, decision maker, query rewriter) that are trained separately, and intermediate "golden" labels for these modules are unavailable or model-dependent, making separate optimization suboptimal.
Approach
A single LLM policy acts as decision maker (retrieve or answer), query rewriter, and answer generator, interacting with a Bing retriever over multiple steps; all roles are jointly optimized with PPO using only exact-match plus F1 answer rewards and a fixed penalty per retrieval, after an SFT warm-up.
Key results
- On Flan-T5-large, SmartRAG averages 33.29 EM / 41.25 F1 across PopQA, AmbigNQ, and HotpotQA, above GPT4 Rewriter + SFT RAG (32.27 / 39.80).
- On LLaMA-2 7B (LoRA), SmartRAG averages 36.27 EM / 43.93 F1, versus 34.54 / 42.49 for GPT4 Rewriter + SFT RAG.
- On OpenBookQA, MedQA-cn and ARC-c, SmartRAG learns not to retrieve (0% retrieval) and keeps accuracy comparable to SFT (66.60 vs 67.20 on OpenBookQA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly optimize the policy and retriever-related tasks with PPO rather than separately optimizing modules | SmartRAG · PopQA, AmbigNQ, and HotpotQA | EM | Helped |
| jointly optimize the policy and retriever-related tasks with PPO rather than separately optimizing modules | SmartRAG · PopQA, AmbigNQ, and HotpotQA | F1 | Helped |
| add a learned query rewriter instead of searching with the original question (ablation) | SmartRAG · PopQA | EM: 39.13 -> 42.50 | Helped |
| add a learned query rewriter instead of searching with the original question (ablation) | SmartRAG · AmbigNQ | EM: 33.41 -> 33.71 | Helped |
| add a learned query rewriter instead of searching with the original question (ablation) | SmartRAG · HotpotQA | F1: 34.04 -> 35.23 | Helped |
| use the jointly optimized PPO answer generator instead of the pre-PPO warm-up generator (ablation) | SmartRAG · PopQA | EM: 40.95 -> 42.50 | Helped |
| use the jointly optimized PPO answer generator instead of the pre-PPO warm-up generator (ablation) | SmartRAG · AmbigNQ | EM: 33.23 -> 33.71 | Helped |
| use the jointly optimized PPO answer generator instead of the pre-PPO warm-up generator (ablation) | SmartRAG · HotpotQA | F1: 35.04 -> 35.23 | Helped |
| initialize PPO from the better warm-up policy using data selected by the base model's answer ability (ablation) | SmartRAG · PopQA, AmbigNQ, and HotpotQA | EM | Helped |
| replace GPT-3.5 with LLaMA-7B to generate the SFT query dataset (ablation) | SmartRAG · PopQA, AmbigNQ, and HotpotQA | F1 | No clear effect |
| increase retrieved document count from K=1 to K=4 (ablation) | SmartRAG · Flan-T5-large evaluation | performance | Helped |
| make retrieval adaptive rather than always retrieve | SmartRAG · OpenBookQA, MedQA-cn, and ARC-c | accuracy | Mixed |
Surprising findings
- GPT-4-rewritten queries sometimes underperform the original question, suggesting reasonable rewrites are not necessarily optimal for a given retriever.
- SKR retrieves for 84.29% of MedQA-cn questions despite the database containing little relevant knowledge, while SmartRAG retrieves 0%.
Limitations
- Retrieval penalty alpha must be tuned, and extreme values drive the policy to always or never retrieve.
- Multi-hop datasets (2WikiMultiHopQA, MuSiQue) are treated as single-step tests, simplifying the task.
- Relies on the Bing search API, so results depend on a commercial retriever and its up-to-date web content.
Experimental setup
Datasets: PopQA AmbigNQ HotpotQA TriviaQA OpenBookQA MedQA-cn ARC-c 2WikiMultiHopQA MuSiQue
Benchmarks: PopQA AmbigNQ HotpotQA TriviaQA 2WikiMultiHopQA MuSiQue OpenBookQA MedQA-cn ARC-c
Metrics: exact match F1 hit rate retrieval percentage accuracy
Baselines: Vanilla RAG SFT RAG GPT4 Rewriter + SFT RAG Self-RAG SKR GPT4 Rewriter + SKR ReAct IRCoT ICL SFT
Builds on: PPO Self-RAG SKR ReAct IRCoT
Closest papers by meaning
- Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards (ICLR 2025 · similarity 0.85)
- C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation (ICML 2025 · similarity 0.85)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.84)
- JADE: Bridging the Strategic-Operational Gap in Dynamic Agentic RAG (ICML 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.