Research map / Multi-agent systems and games / LLM agents

Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning

NeurIPS 2025 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Models a four-module RAG pipeline (query rewriter, fixed retriever, selector, generator) as a cooperative multi-agent RL problem and jointly fine-tunes the LLM-based agents with MAPPO using a shared F1 reward.

Problem

Components of multi-module RAG pipelines are optimized separately with SFT, leading to misalignment with the goal of accurate final answers; prior RL-based RAG methods handle only two components or optimize modules in isolation.

Approach

Treat query rewriter, selector, and generator as cooperating RL agents sharing one LLM (parameter sharing) and a shared reward equal to the final-answer F1, optimized jointly with MAPPO after an SFT warm start, with per-agent format/length penalties.

Key results

• On HotpotQA with Contriever, MMOA-RAG reaches 39.15 Acc / 36.15 EM / 48.29 F1, versus 38.03 / 33.93 / 46.32 for the best baseline (Rewrite-Retrieve-Read).
• MAPPO joint training improves over the SFT warm start by about 3 F1 points on HotpotQA and 2WikiMultihopQA and about 2 F1 points on AmbigQA across three module configurations.
• MMOA-RAG attains the best F1 across HotpotQA, 2WikiMultihopQA and AmbigQA with Contriever, and also leads on BGE and E5 retrievers in most settings.

Measured effects reported in the paper

Surprising findings

• Removing the Selector from joint optimization (MMOA-RAG w/o S) performed best among the ablations, suggesting the generator learns denoising under MAPPO.
• Vanilla RAG without training performs about the same as LLM without retrieval, indicating Llama-3-8B-Instruct struggles to use retrieved documents without RAG-specific training.

Limitations

• Retriever is fixed and not optimized as an agent.
• Reward relies solely on answer F1, ignoring cost and latency of modules.
• Selector SFT labels are built with a heuristic keyword-overlap method that may be noisy.

Experimental setup

Datasets: HotpotQA 2WikiMultihopQA AmbigQA Rewrite-Retrieve-Read query rewriting data

Benchmarks: HotpotQA 2WikiMultihopQA AmbigQA

Metrics: Accuracy Exact Match F1

Baselines: LLM w/o RAG Vanilla RAG w/o train Vanilla RAG w SFT SELF-RAG RetRobust Rewrite-Retrieve-Read BGM RAG-DDR

Builds on: MAPPO PPO Rewrite-Retrieve-Read BGM RAG-DDR

Closest papers by meaning

• C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation (ICML 2025 · similarity 0.92)
• Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.89)
• RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards (ICLR 2025 · similarity 0.88)
• MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.88)
• MMRAG-RFT: Two-stage Reinforcement Fine-tuning for Explainable Multi-modal Retrieval-augmented Generation (AAAI 2026 · similarity 0.88)