Research map / Large language models / Retrieval and knowledge
LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LaRA, a 2326-case benchmark over novels, financial reports and academic papers at 32k and 128k contexts, comparing RAG against long-context LLMs across 11 models and four QA task types.
Problem
Existing comparisons of retrieval-augmented generation (RAG) versus long-context (LC) LLMs are inconclusive because of flawed benchmarks (short contexts, truncation, entity replacement errors, unreliable metrics, homogeneous queries).
Approach
A benchmark built from recent naturally occurring long documents near 32k and 128k token lengths, with four task categories (location, reasoning, comparison, hallucination detection), GPT-4o-generated QA pairs with human validation, and LLM-as-judge scoring, used to systematically compare RAG and LC routing factors such as model size, context length, task type and chunk settings.
Key results
- At 128k context, RAG outperformed LC by 3.68% on average across models, while at 32k LC led by 2.40%.
- Proprietary models (GPT-4o, Claude-3.5-sonnet, Gemini-1.5-pro) at 128k all rank top using long-context, and Mistral-Nemo-12B at 128k drops to 15.21% overall with LC versus 53.33% with RAG.
- On comparison tasks LC beats RAG by 15.22% at 32k and 14.30% at 128k; on hallucination detection RAG beats LC by 10.38% at 32k and 22.36% at 128k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAG instead of long context at 32k | Evaluated models on LaRA · LaRA 32k overall | accuracy: 2.40% | Hurt |
| Use RAG instead of long context at 128k | Evaluated models on LaRA · LaRA 128k overall | accuracy: 3.68% | Helped |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 32k location | accuracy: 0.08 | No clear effect |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 128k location | accuracy: -5.25 | Helped |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 32k reasoning | accuracy: 4.66 | Hurt |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 128k reasoning | accuracy: -1.39 | Helped |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 32k comparison | accuracy: 15.22 | Hurt |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 128k comparison | accuracy: 14.30 | Hurt |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 32k hallucination | accuracy: -10.38 | Helped |
| Use RAG instead of long context | Evaluated models on LaRA · LaRA 128k hallucination | accuracy: -22.36 | Helped |
| Increase retrieved chunk quantity (ablation) | Qwen-2.5-72B-Instruct RAG · LaRA | accuracy | Helped |
| Increase retrieved chunk quantity beyond the intermediate peak (ablation) | Qwen-2.5-7B-Instruct RAG · LaRA | accuracy | Hurt |
Surprising findings
- Qwen-2.5-7B with segmented input and voting outperformed GPT-4o on the InfiniteBench En.MC task (87.9 vs 76.0), suggesting prior truncation-based results were misleading.
- GPT-4o reaches only 56.34% on hallucination detection at 128k, below several smaller models, indicating large models are not better at refusing unanswerable questions.
Limitations
- Evaluation relies on GPT-4o as judge and QA generator, which may introduce bias, though Cohen's Kappa checks were on only 100 samples per task.
- Benchmark covers only three context types (novels, financial statements, academic papers) and a single RAG configuration (600-token chunks, 5 chunks, GTE-large-en-v1.5 plus BM25).
Experimental setup
Datasets: Project Gutenberg novels annualreports.com financial statements (2024) arXiv papers (2024)
Benchmarks: LaRA
Metrics: accuracy Cohen's Kappa average gap between LC and RAG
Baselines: GPT-4o GPT-4o-mini Claude-3.5-sonnet Gemini-1.5-pro Llama-3.1-8B-Instruct Llama-3.2-3B-Instruct Llama-3.3-70B-Instruct Qwen-2.5-7B-Instruct Qwen-2.5-72B-Instruct Mistral-Nemo-12B
Builds on: in-context learning BM25 Needle in a Haystack InfiniteBench
Closest papers by meaning
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.93)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.90)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.90)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.90)
- One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models (AAAI 2025 · similarity 0.90)
- ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.