Research map / Large language models / Retrieval and knowledge
Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReasonRAG, a process-supervised agentic RAG method that uses MCTS with a shortest-path reward estimate to build the RAG-ProGuide process-preference dataset and trains Qwen2.5-7B-Instruct with DPO, reaching better QA scores than Search-R1 with 5k queries.
Problem
Outcome-supervised RL for agentic RAG (e.g., Search-R1) suffers from low exploration efficiency, gradient conflict, sparse rewards, and needs large training sets (90k queries).
Approach
Assign fine-grained process-level rewards to intermediate RAG steps (query generation, evidence extraction, answer generation) via a Monte Carlo estimate with a step-decay penalty (Shortest Path Reward Estimation) over MCTS-explored trees, and optimize the policy with process-level preference optimization (DPO).
Key results
- ReasonRAG (5k queries) averages 34.4 EM and 42.3 F1 across five benchmarks, versus Search-R1 (90k queries) at 32.8 EM and 40.7 F1.
- On HotpotQA, ReasonRAG reaches 48.9 F1 versus 47.0 for Search-R1.
- Outcome-RL variant trained on 5k queries reaches only 21.3 EM average, far below the process-RL variant's 34.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use process-level rewards and train with 5k queries instead of outcome-level rewards with 90k queries | ReasonRAG vs. Search-R1 · Avg. across five benchmarks | EM: 32.8 -> 34.4 | Helped |
| use process-level rewards and train with 5k queries instead of outcome-level rewards with 90k queries | ReasonRAG vs. Search-R1 · Avg. across five benchmarks | F1: 40.7 -> 42.3 | Helped |
| add process-level preference optimization (ablation) | ReasonRAG · Avg. across five benchmarks | EM: 22.0 -> 34.5 | Helped |
| add process-level preference optimization (ablation) | ReasonRAG · Avg. across five benchmarks | F1: 32.8 -> 42.3 | Helped |
| replace process-level preference optimization with SFT on preferred responses (ablation) | ReasonRAG · Bamboogle | EM: 36.0 -> 17.6 | Hurt |
| replace process-level rewards with outcome-level rewards using the same 5k queries (ablation) | ReasonRAG · Avg. across five benchmarks | F1: 42.3 -> 28.9 | Hurt |
| replace process-level rewards with outcome-level rewards and increase training data to 10k queries (ablation) | ReasonRAG · Avg. across five benchmarks | F1: 42.3 -> 39.9 | Hurt |
| increase retrieval iterations | ReasonRAG · PopQA, 2WikiMultiHopQA, and HotpotQA | EM | Helped |
| increase retrieved documents from top-1 to top-3 | ReasonRAG · all evaluated datasets | EM | Helped |
| increase retrieved documents from top-3 to top-5 | ReasonRAG · PopQA and HotpotQA | EM | No clear effect |
| increase retrieved documents from top-3 to top-5 | ReasonRAG · 2WikiMultiHopQA | EM | Helped |
Surprising findings
- SFT on preferred responses overfits multi-hop paths and generalizes poorly to Bamboogle.
- Outcome-RL with 5k queries underperforms the base model's SFT-free variant on average EM, indicating outcome RL needs much more data.
Limitations
- Collecting process-level annotations via MCTS is more costly in time than outcome supervision.
- Process supervision relies on an LLM judge and a single backbone (Qwen2.5-7B-Instruct) evaluated.
Experimental setup
Datasets: PopQA HotpotQA 2WikiMultiHopQA Bamboogle MuSiQue Wikipedia 2018 dump
Benchmarks: PopQA HotpotQA 2WikiMultiHopQA Bamboogle MuSiQue
Metrics: Exact Match F1
Baselines: Naïve Generation Standard RAG FLARE Self-RAG AdaptiveRAG Iter-Retgen IRCoT RECOMP LongLLMLingua Selective-Context Search-o1 AutoRAG
Builds on: Search-R1 Direct Preference Optimization Monte Carlo Tree Search FlashRAG
Closest papers by meaning
- Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation (ICML 2025 · similarity 0.88)
- RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards (ICLR 2025 · similarity 0.88)
- SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedback (ICLR 2025 · similarity 0.88)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.87)
- SD-Search: Hindsight Self-Distillation for Search-Augmented Reasoning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.