Research map / Large language models / Retrieval and knowledge
Retrieval meets Long Context Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares retrieval augmentation against positional-interpolation context extension (4K to 16K/32K) for GPT-43B and Llama2-70B on long-context QA and summarization tasks, and shows that combining both gives the best results.
Problem
Whether extending the context window of LLMs or augmenting them with retrieval works better for long-context downstream tasks, and whether the two can be combined.
Approach
Systematic controlled comparison of retrieval-augmentation against positional-interpolation context extension on instruction-tuned 43B and 70B LLMs, finding a 4K model with retrieval rivals a 16K model without it, and that retrieval further improves long-context models.
Key results
- Llama2-70B with 4K context and Contriever retrieval (top-5) reaches avg 36.02 vs 36.78 for the non-retrieval 16K model.
- Best model Llama2-70B-32k-ret (Dragon, top-5) averages 39.60 on seven tasks, vs 37.36 without retrieval and above GPT-3.5-turbo-16k on the four ZeroSCROLLS-style averages (43.0 vs 42.4 for Avg-4*).
- Retrieval-augmented GPT-43B at 4K improves average from 26.44 to 29.32 on the seven datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add retrieval (top 5 chunks) to the 4k model | GPT-43B · Seven long-context tasks | average score: 26.44 -> 29.32 | Helped |
| Add retrieval (top 5 chunks) to the 16k model | GPT-43B · Seven long-context tasks | average score: 29.45 -> 29.65 | Helped |
| Add retrieval (top 5 chunks) to the 4k model | Llama2-70B · Seven long-context tasks | average score: 31.61 -> 36.02 | Helped |
| Add retrieval (top 5 chunks) to the 16k model | Llama2-70B · Seven long-context tasks | average score: 36.78 -> 37.23 | Helped |
| Add retrieval (top 5 chunks) to the 32k model | Llama2-70B · Seven long-context tasks | average score: 37.36 -> 39.60 | Helped |
| Add retrieval to the 32k model | Llama2-7B · Seven long-context tasks | average score: 28.20 -> 27.63 | Hurt |
| Use Dragon instead of no retrieval (ablation) | Llama2-70B, 4k · Seven long-context tasks | average score: 31.61 -> 35.73 | Helped |
| Use Contriever instead of no retrieval (ablation) | Llama2-70B, 4k · Seven long-context tasks | average score: 31.61 -> 36.02 | Helped |
| Use OpenAI embeddings instead of no retrieval (ablation) | Llama2-70B, 4k · Seven long-context tasks | average score: 31.61 -> 35.79 | Helped |
| Increase retrieved chunks from 5 to 10 (ablation) | Llama2-70B, 16k · Seven long-context tasks | average score: 37.23 -> 38.31 | Helped |
| Increase retrieved chunks from 5 to 20 (ablation) | Llama2-70B, 16k · Seven long-context tasks | average score: 37.23 -> 36.61 | Hurt |
| Increase retrieved chunks from 5 to 20 (ablation) | Llama2-70B, 32k · Seven long-context tasks | average score: 39.60 -> 38.38 | Hurt |
Surprising findings
- Retrieval-augmented 4K Llama2-70B nearly matches a 16K context-extended model while using much less computation.
- Adding more retrieved chunks (top-10 or top-20) often hurts performance relative to top-5, suggesting distraction or lost-in-the-middle effects.
Limitations
- GPT-3.5-turbo results rely on a black-box API with unknown implementation and size.
- Findings differ from prior work (LongBench) using smaller models, so conclusions may depend on model scale.
- Adding more retrieved chunks does not consistently help, and the cause is not resolved.
Experimental setup
Datasets: The Pile Soda ELI5 FLAN Open Assistant Dolly QMSum Qasper NarrativeQA QuALITY HotpotQA MuSiQue
Benchmarks: ZeroSCROLLS (QMSum, Qasper, NarrativeQA, QuALITY) LongBench (HotpotQA, MuSiQue, MultiFieldQA-en, Trec, SAMSum)
Metrics: ROUGE geometric mean exact match F1 average score
Baselines: GPT-3.5-turbo-16k Davinci-003 GPT-3.5-turbo-4k Llama2-7B-32k
Builds on: positional interpolation Dragon retriever Contriever FlashAttention
Closest papers by meaning
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.93)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.92)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.91)
- Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.91)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.90)
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.