Research map / Large language models / Retrieval and knowledge
RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Differentiable Data Rewards (DDR), which trains each agent in a RAG system with DPO on preference pairs whose rewards come from rolling out the downstream system, yielding RAG-DDR.
Problem
SFT-based optimization of RAG modules overfits training signals and ignores differing data preferences between agents (knowledge refinement and generation) in a RAG system.
Approach
Each agent samples perturbed outputs; the downstream sub-system is rolled out to score each output, and the agent is trained with a DPO loss on the highest- vs lowest-reward pairs, so that modules are aligned to the whole system's final quality score.
Key results
- On MiniCPM-2.4B, RAG-DDR (w/ 2-Round) reaches 47.6 NQ, 83.6 TriviaQA, 29.8 MARCO QA, 33.2 HotpotQA, 18.2 WoW, outperforming RA-DIT (41.8 NQ, 78.6 TriviaQA).
- On Llama3-8B, RAG-DDR (w/ 2-Round) achieves 52.1 NQ, 89.6 TriviaQA and 39.0 HotpotQA, above RA-DIT (46.2, 87.4, 34.9).
- Training only the generation module gives most of the gain, with knowledge refinement training adding small further improvements.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAG-DDR instead of the strongest non-DDR baseline | MiniCPM-2.4B RAG · NQ | Accuracy: 41.8 -> 47.6 | Helped |
| Use RAG-DDR instead of the strongest non-DDR baseline | MiniCPM-2.4B RAG · MARCO QA | Rouge-L: 19.6 -> 29.8 | Helped |
| Use RAG-DDR instead of the strongest non-DDR baseline | Llama3-8B RAG · HotpotQA | Accuracy: 34.9 -> 39.0 | Helped |
| Add DDR training to both modules, rather than training only the generation module (ablation) | MiniCPM-2.4B RAG · HotpotQA | Accuracy: 31.2 -> 32.5 | Helped |
| Add DDR training to both modules, rather than training only the generation module (ablation) | Llama3-8B RAG · NQ | Accuracy: 50.2 -> 50.7 | Helped |
| Add the knowledge refinement module to zero-shot Vanilla RAG without it (ablation) | MiniCPM-2.4B RAG · HotpotQA | Accuracy: 24.9 -> 26.7 | Helped |
| Optimize the generation module with DDR instead of independently tuning the modules (ablation) | MiniCPM-2.4B RAG · NQ | Accuracy: 46.6 -> 47.0 | Helped |
| Optimize the knowledge refinement module before the generation module instead of the reverse order (ablation) | MiniCPM-2.4B RAG · NQ | Accuracy: 47.4 vs 47.0 | No clear effect |
| Use RAG-DDR instead of Vanilla RAG | Qwen2.5-14B-Instruct RAG · T-REx | Accuracy: 37.3 -> 44.6 | Helped |
| Use DDR rather than RA-DIT training (ablation) | MiniCPM-2.4B generation module · NQ | Accuracy: 40.9 -> 46.8 | Helped |
| Use DDR rather than RA-DIT ground-truth SFT (ablation) | Llama3-8B generation module · HotpotQA | Accuracy: 34.9 -> 36.9 | Helped |
| Add summarization and knowledge refinement training to generation-module DDR training (ablation) | Three-agent MiniCPM-2.4B RAG · NQ | Accuracy: 46.6 -> 47.9 | Helped |
Surprising findings
- RA-DIT trained with only positive labels performs worse than RA-DIT trained on ground-truth labels, suggesting positive-only SFT is insufficient.
- Training only the knowledge refinement module yields limited gains compared with optimizing the generation module.
Limitations
- Gains on some tasks (e.g., T-REx, MARCO QA) are not uniformly better than baselines; RA-DIT leads on T-REx with Llama3-8B.
- Evaluation relies on string-matching metrics (Accuracy, Rouge-L, F1) that may penalize long responses, as the InstructRAG comparison suggests.
- Retriever is not fine-tuned during reproduction, and the rollout process requires sampling many outputs per query, which is computationally costly.
Experimental setup
Datasets: Commonsense QA Math QA Web Questions Wiki QA Yahoo! Answers QA MARCO QA Algebra QA with Rationales Explanations for CommonsenseQA Grade School Math 8K StrategyQA MS MARCO 2.1
Benchmarks: NQ TriviaQA MARCO QA HotpotQA T-REx WoW
Metrics: Accuracy Rouge-L F1
Baselines: LLM w/o RAG Vanilla RAG REPLUG Self-RAG RA-DIT InstructRAG Filco
Builds on: DPO RA-DIT Self-RAG REPLUG
Closest papers by meaning
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.92)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.90)
- Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems (ICLR 2025 · similarity 0.90)
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.90)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.89)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.