Research map / Multi-agent systems and games / LLM agents
C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-3PO, a lightweight 0.5B/1.5B multi-agent proxy trained with MARL and tree-structured rollouts to align frozen retrievers and LLMs for RAG.
Problem
Aligning independently developed retrievers and LLMs in RAG without modifying either component, while optimizing the full pipeline rather than single subtasks like query rewriting or reranking.
Approach
A single small proxy model plays three agents (Reasoning Router, Information Filter, Decision Maker) that mediate between a frozen retriever and frozen LLM; it is trained with PPO using Monte Carlo credit assignment computed from a tree-structured rollout that explores all reasoning strategies per question.
Key results
- C-3PO with Qwen2-1.5B proxy reaches 63.53% average on in-domain QA vs 51.35% for Standard RAG and 54.33% for SlimPLM with Qwen2-72B server.
- On OOD settings with Google search, C-3PO averages 59.28% vs 55.06% for the best baseline (Standard RAG), a 4.22-point gain.
- On HLE text-only questions, C-3PO averages 5.79% vs 4.27% for its Qwen2.5-72B base LLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use C-3PO with a Qwen2-1.5B proxy instead of the strongest listed baseline | RAG with Qwen2-72B · 2Wiki | accuracy: 49.7 -> 65.2 | Helped |
| Use C-3PO with a Qwen2-1.5B proxy instead of the strongest listed baseline | RAG with Qwen2-72B · HQA | accuracy: 55.8 -> 69.0 | Helped |
| Use C-3PO with a Qwen2-1.5B proxy instead of the strongest listed baseline | RAG with Qwen2-72B · Musique | accuracy: 41.3 -> 54.2 | Helped |
| Use C-3PO with a Qwen2-1.5B proxy instead of the strongest listed baseline | RAG with Qwen2-72B · NQ | accuracy: 62.4 -> 65.9 | Helped |
| Use C-3PO with a Qwen2-1.5B proxy instead of the strongest listed baseline | RAG with Qwen2-72B · PopQA | accuracy: 40.9 -> 44.8 | Helped |
| Use C-3PO with a Qwen2-1.5B proxy instead of the strongest listed baseline | RAG with Qwen2-72B · TQA | accuracy: 78.9 -> 82.1 | Helped |
| Add reinforcement learning to the C-3PO-ICL framework | C-3PO proxy · 2Wiki | accuracy: 54.1 -> 65.2 | Helped |
| Add reinforcement learning to the supervised warm-up model (ablation) | C-3PO · 2Wiki, Musique, PopQA, and TQA | accuracy: +1.6%, +4.5%, +1.7%, and +2.0% | Helped |
| Remove tree-structured rollout and Monte Carlo credit assignment, using a single trajectory and system-level rewards (ablation) | C-3PO RL training · Six in-domain datasets | accuracy | Hurt |
| Force the Direct Answering strategy for every question instead of adaptive strategy selection (ablation) | C-3PO · 2Wiki, PopQA, FQA, and M-RAG | accuracy: 2Wiki: 65.2 -> 41.6; PopQA: 44.8 -> 24.3; FQA: 72.8 -> 58.2; M-RAG: 50.0 -> 42.2 | Hurt |
| Force the Single-pass Retrieval strategy for every question instead of adaptive strategy selection (ablation) | C-3PO · 2Wiki, PopQA, FQA, and M-RAG | accuracy: 2Wiki: 65.2 -> 64.7; PopQA: 44.8 -> 44.6; FQA: 72.8 -> 68.8; M-RAG: 50.0 -> 48.3 | Mixed |
| Force the Multi-Step Planning strategy for every question instead of adaptive strategy selection (ablation) | C-3PO · 2Wiki, PopQA, FQA, and M-RAG | accuracy: 2Wiki: 65.2 -> 65.5; PopQA: 44.8 -> 46.9; FQA: 72.8 -> 74.8; M-RAG: 50.0 -> 50.4 | Mixed |
Surprising findings
- Removing tree-structured rollout made RL unstable, sometimes degrading below the supervised warm-up model.
- Model-level in-context variant C-3PO-ICL with a 72B proxy scored lower than the RL-trained 1.5B proxy on multi-hop datasets.
Limitations
- Experiments are limited to QA tasks with LLM-judged answers and a Wikipedia/Google retriever setup.
- Inference cost (4.8s per question) is higher than standard RAG (3.6s), and ICL variant with 72B proxy is much slower (10.7s).
Experimental setup
Datasets: Natural Questions PopQA TriviaQA 2WikiMultiHopQA Musique HotpotQA FreshQA MultiHop-RAG
Benchmarks: 2WikiMultiHopQA HotpotQA Musique Natural Questions PopQA TriviaQA FreshQA MultiHop-RAG Humanity's Last Exam
Metrics: accuracy (LLM-judged) average accuracy inference latency
Baselines: Direct Standard RAG REPLUG Self-RAG InstructRAG Auto-RAG Reranker QueryRewrite SKR-KNN SlimPLM
Builds on: PPO Self-RAG REPLUG SlimPLM Auto-RAG
Closest papers by meaning
- Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.89)
- Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents (ICLR 2026 · similarity 0.88)
- JADE: Bridging the Strategic-Operational Gap in Dynamic Agentic RAG (ICML 2026 · similarity 0.88)
- RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.