Research map / Large language models / Retrieval and knowledge
MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaFeRw, a PPO-based query rewriter for conversational RAG whose dense rewards come from three reward models (gold-document similarity, retrieved-document ranking vs. ground truth, generation ROUGE) plus rewrite ROUGE against manual rewrites.
Problem
Query rewriting for multi-turn conversational RAG is weakly coupled to generation quality, and RL with sparse generation-based rewards (end-to-end RAG inference) is unstable and slow.
Approach
Replace sparse end-to-end generation rewards with multi-aspect dense rewards: three reward models trained (RLAIF-style, Bradley-Terry on preference pairs) to predict retrieval and generation metrics, combined with rewrite-vs-manual ROUGE, and optimize a T5 rewriter with PPO.
Key results
- On QReCC, MaFeRw reaches ROUGE-1 37.05 and MRR 0.5032, versus 35.73 / 0.4884 for the T5-base rewriter and 31.98 / 0.4729 for raw RAG.
- On TopiOCQA, MaFeRw reaches ROUGE-1 23.97 and MRR 0.3802, versus 21.46 / 0.3015 for the T5-base rewriter.
- Reward models classify chosen/rejected pairs with accuracy 0.7277-0.8927 across the three metrics on QReCC and TopiOCQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaFeRw multi-aspect feedback instead of T5-base rewriting | RAG on QReCC · QReCC | ROUGE-1: 35.73 -> 37.05 | Helped |
| Use MaFeRw multi-aspect feedback instead of ConvGQR | RAG on QReCC · QReCC | MRR: 0.4974 -> 0.5032 | Helped |
| Use MaFeRw multi-aspect feedback instead of RL-base | RAG on TopiOCQA · TopiOCQA | ROUGE-1: 22.94 -> 23.97 | Helped |
| Use MaFeRw multi-aspect feedback instead of RL-base | RAG on TopiOCQA · TopiOCQA | MRR: 0.3717 -> 0.3802 | Helped |
| Apply the QReCC-trained MaFeRw rewriter instead of T5-base | RAG on WSDM@24 Multi-Doc QA · WSDM@24 Multi-Doc QA | ROUGE-1: 39.80 -> 41.01 | Helped |
| Use only m_d+ feedback rather than the combined feedback (ablation) | MaFeRw on QReCC · QReCC | ROUGE-1: 37.04 -> 36.10 | Hurt |
| Use only m_D feedback rather than the combined feedback (ablation) | MaFeRw on QReCC · QReCC | ROUGE-1: 37.04 -> 35.69 | Hurt |
| Use only m_G feedback rather than the combined feedback (ablation) | MaFeRw on QReCC · QReCC | ROUGE-1: 37.04 -> 36.69 | Hurt |
| Use only m_q feedback rather than the combined feedback (ablation) | MaFeRw on QReCC · QReCC | ROUGE-1: 37.04 -> 36.88 | Hurt |
| Use only m_d+ feedback rather than the combined feedback (ablation) | MaFeRw on TopiOCQA · TopiOCQA | ROUGE-1: 23.97 -> 21.27 | Hurt |
| Use only m_D feedback rather than the combined feedback (ablation) | MaFeRw on TopiOCQA · TopiOCQA | ROUGE-1: 23.97 -> 21.86 | Hurt |
| Use only m_G feedback rather than the combined feedback (ablation) | MaFeRw on TopiOCQA · TopiOCQA | MRR: 0.3802 -> 0.3570 | Hurt |
Surprising findings
- Single-metric RL variants often underperform the T5-base rewriter on some metrics, e.g., the RL-base rewriter scores lower than T5-base on QReCC.
- ConvGQR's query expansion improves MRR but yields lower ROUGE-1 than the T5-base rewriter on TopiOCQA (17.05 vs 21.46).
Limitations
- Gains over baselines are modest on some metrics, and single-metric variants are unstable across datasets.
- Experiments use a single generator (Llama-2-13b-chat) and one retriever setup; the authors filtered document collections due to hardware limits.
- Reward model accuracy on the ranking and generation metrics is only 0.73-0.85 on some splits.
Experimental setup
Datasets: QReCC TopiOCQA WSDM@24 Multi-Doc QA
Benchmarks: QReCC TopiOCQA WSDM@24 Multi-Doc QA
Metrics: ROUGE-1 ROUGE-L BLEU METEOR MRR ROUGE-2
Baselines: raw RAG T5-base rewriter RL-base rewriter ConvGQR
Builds on: RLAIF Bradley-Terry model PPO T5 ConvGQR
Closest papers by meaning
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.89)
- RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting (AAAI 2024 · similarity 0.87)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.87)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.87)
- Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.