Research map / Large language models / Retrieval and knowledge
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that adding more retrieved passages to long-context LLM RAG can hurt (inverted-U with strong retrievers) due to hard negatives, and proposes retrieval reordering plus implicit and reasoning-augmented RAG fine-tuning to improve robustness.
Problem
Long-context LLMs enable many more retrieved passages in RAG, but it is unclear whether more retrieved context improves performance, given hard negatives from retrievers.
Approach
Analyzes how the number and difficulty of retrieved hard negatives affect long-context LLMs; proposes a training-free retrieval reordering that places high-score passages at the start and end of the context, and fine-tunes LLMs on RAG data (with and without intermediate reasoning labels) to improve robustness to noisy retrieved context.
Key results
- With the strong e5 retriever, RAG accuracy on NQ rises then declines or plateaus as passages increase, while BM25 shows steadier gains.
- Retrieval reordering gives negligible gains with few passages but consistently outperforms original ordering with many passages on NQ and PopQA.
- RAG fine-tuned Gemma-2-9B-Base and Mistral-Nemo-12B-Base outperform chat models with RAG across unseen datasets and degrade less as passage count grows; intermediate reasoning adds further gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of retrieved passages using e5 | Long-context LLMs in RAG · NQ | answer performance | Mixed |
| Increase the number of retrieved passages using BM25 | Long-context LLMs in RAG · NQ | answer performance | Mixed |
| Add hard negative passages from stronger retrievers | Long-context LLMs · NQ | answer accuracy | Hurt |
| Increase the number of hard negative passages | Long-context LLMs · NQ | answer accuracy | Hurt |
| Reorder retrieved passages by retrieval score, placing high-scoring passages near the beginning and end | Long-context RAG · NQ and PopQA | answer performance | Helped |
| Fine-tune on queries with retrieved passages and answer targets (implicit RAG FT) | Gemma-2-9B-Base, Mistral-Nemo-12B-Base, and Gemini-1.0-Pro · Unseen RAG datasets | performance | Helped |
| Add an intermediate reasoning step that identifies relevant passages during RAG fine-tuning | RAG-tuned LLMs · Unseen RAG datasets | performance | Helped |
| Train on a mixed distribution of NQ, WoW, FEVER, and MMLU instead of a single source (ablation) | RAG-tuned LLM · Unseen RAG tasks | generalization performance | Helped |
| Train with passages from a mix of BM25 and e5 instead of one retriever (ablation) | Gemma-2-9B-Base · Seen and unseen retrievers | performance | Helped |
| Train with the maximum context length (40 passages) instead of shorter or dynamic passage counts (ablation) | Gemma-2-9B-Base · NQ | performance across inference retrieval sizes | Helped |
| Use RAG-specific fine-tuning instead of direct fine-tuning on question-answer pairs without retrieved context (ablation) | RAG-tuned LLMs · Unseen RAG datasets | performance | Helped |
Surprising findings
- Increasing retrieved passages does not consistently improve long-context RAG; a stronger retriever (e5) yields a sharper degradation than BM25.
- Hard negatives from a stronger retriever can mislead LLMs more than those from a weaker retriever, and random-negative benchmarks like needle-in-the-haystack may not capture this.
Limitations
- Gemini-based results rely on closed APIs and cannot be fully reproduced.
- Reasoning labels are generated by Gemini-1.5-Pro, and inference with reasoning is costly.
- Analysis focuses on single-hop/QA datasets with Wikipedia retrieval; multi-golden-passage settings are excluded from the controlled study.
Experimental setup
Datasets: Natural Questions Wizard of Wikipedia FEVER MMLU PopQA Wikipedia 2018 dump
Benchmarks: Natural Questions PopQA TriviaQA HotpotQA 2WikiMultiHopQA WebQuestions Bamboogle ASQA T-REx Zero-shot RE
Metrics: accuracy recall@k precision@k
Baselines: Chat model with RAG (Gemma-2-9B-Chat) Mistral-Nemo-12B-Instruct w. RAG Gemini-1.0-Pro w. RAG Direct FT (no retrieval) BM25 Contriever BGE
Builds on: lost-in-the-middle RetRobust RA-DIT RAFT InstructRAG
Closest papers by meaning
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.94)
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.93)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.92)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.92)
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.92)
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.