Research map / Large language models / Retrieval and knowledge
Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks
Xinxi Lyu, Michael Duan, Rulin Shao, Pang Wei Koh, Sewon Min
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CompactDS retrieval (k=10) | Llama 3.1 8B Instruct · MMLU Pro | accuracy: 39.8 -> 53.1 | Improved |
| add CompactDS retrieval (k=10) | Llama 3.1 8B Instruct · Average over benchmarks | accuracy: 48.3 -> 55.1 | Improved |
| add exact search with GRIT after ANN retrieval (ablation) | CompactDS-ANN only (k=10) · Average over benchmarks | accuracy: 53.8 -> 55.1 | Improved |
| add exact search with Contriever (same model as ANN) (ablation) | CompactDS-ANN only · Average over benchmarks | accuracy: 53.8 -> 53.6 | No clear effect |
| add LM reranking (ablation) | CompactDS · Average over benchmarks | accuracy: 55.1 -> 56.0 | Mixed |
| use CompactDS instead of a single Wikipedia (DPR) datastore | Llama 3.1 8B Instruct · Average over benchmarks | accuracy: 49.0 -> 54.1 | Improved |
| compress index from 456GB to 125GB (64 subquantizers) (ablation) | CompactDS · Average over benchmarks | accuracy: 55.1 -> 54.1 | No clear effect |
| use oracle reranking of 100 retrieved passages | CompactDS-ANN · Average over benchmarks (6 columns) | accuracy: 63.0 -> 71.2 | Improved |
| scale generator to Llama 3.3 70B Instruct and add CompactDS | Llama 3.3 70B Instruct · Average over benchmarks | accuracy: 68.8 -> 71.2 | Improved |
| add CompactDS to 70B model | Llama 3.3 70B Instruct · GPQA Physics | accuracy: 64.2 -> 62.0 | Hurt |
| add CompactDS retrieval | Mistral 7B Instruct · Average over benchmarks | accuracy: 37.1 -> 42.6 | Improved |
| add decontamination of search results (ablation) | Search engine pipeline · MMLU Pro | accuracy: 46.4 -> 42.8 | Hurt |
Closest papers by meaning
- BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (ICLR 2025 · similarity 0.91)
- Retriever-Free Retrieval-Augmented Reasoning via Corpus-Traversing MCTS (NeurIPS 2026 · similarity 0.89)
- Retrieval is Not Enough: Enhancing RAG through Test-Time Critique and Optimization (NeurIPS 2025 · similarity 0.89)
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.88)
- FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (NeurIPS 2025 · similarity 0.87)
- Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
