Research map / Large language models / Retrieval and knowledge
Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HoloBench, an automatically generated benchmark that verbalizes Spider text-to-SQL database rows into long text contexts to evaluate long-context LMs on holistic (aggregation-heavy) reasoning, varying context length, information amount/density, position, and query type/difficulty.
Problem
Long-context language models are mainly evaluated on localized retrieval (needle-in-a-haystack), not on holistic reasoning that aggregates and filters information scattered across massive document collections.
Approach
Convert text-to-SQL database rows into natural-language text via templates, mixing relevant and distractor rows at controlled lengths and positions, so gold answers are computed automatically by executing SQL, enabling scalable evaluation of aggregation, join, comparison, ranking, and max/min queries.
Key results
- Llama-3.1-405B drops from 64.0% at 4k to 44.8% at 64k under constant 2k information, but to 30.2% at 64k under 50% density, indicating information amount matters more than length.
- Max/Min queries are consistently easiest while Aggregation queries degrade most with longer context; GPT-4o has highest wAvg (inc) at 55.0 vs Llama-3.1-405B 53.5 in the constant-amount setting.
- Llama-3.1-8B reaches only 1.1% accuracy at 64k with 50% density, and RAG (bge-large-en-v1.5) beats vanilla LCLMs only at 4k for GPT-4o and at contexts beyond 16k for Llama-3.1-8B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase relevant information with context length under constant 50% information density | Llama-3.1-405b · HoloBench, 4k to 64k context | accuracy: 64.0 -> 30.2 | Hurt |
| Increase context length while keeping relevant information fixed at 2k tokens | Llama-3.1-405b · HoloBench, constant information amount (2k), 4k to 64k context | accuracy: 64.0 -> 44.8 | Hurt |
| Use a retrieval model to select relevant context for short inputs | GPT-4o and Llama-3.1-8b · HoloBench, 4k context | accuracy | Helped |
| Use RAG instead of the vanilla model for longer contexts | Llama-3.1-405b and GPT-4o · HoloBench, contexts longer than 4k | accuracy | Hurt |
| Use RAG rather than the vanilla model for contexts exceeding 16k tokens | Llama-3.1-8b · HoloBench, contexts exceeding 16k | accuracy | Helped |
| Increase retrieved tokens from 2k to 32k when relevant information occupies half of a 64k context (ablation) | RAG with Llama-3.1-405b · HoloBench, 64k context, 50% information density | accuracy: 21.70 -> 37.34 | Helped |
| Use all 64k tokens instead of retrieving 32k tokens when all context is relevant (ablation) | RAG with Llama-3.1-405b · HoloBench, 64k context, 100% information density | accuracy: 38.69 -> 39.06 | Helped |
| Add chain-of-thought prompting instead of answering directly (ablation) | LCLMs · HoloBench, all context lengths and query types | accuracy | Helped |
| Cluster relevant information rather than distribute it uniformly | LCLMs · HoloBench, information positioning | accuracy | Helped |
| Place relevant information at the end or bimodally rather than at the beginning or middle | GPT-4o · HoloBench, information positioning | accuracy | Mixed |
| Ask a Max/Min query rather than other query types | LCLMs · HoloBench, query types across context lengths | accuracy | Helped |
| Require aggregation of multiple information pieces, especially at longer context lengths | LCLMs · HoloBench, Aggregation queries | accuracy | Hurt |
Surprising findings
- Reasoning-focused models (o1-mini, o3-mini, DeepSeek-R1, Gemini Flash Thinking) degrade significantly with longer contexts despite strong short-context performance.
- Max/Min queries are easier for LCLMs even though they are challenging for RAG systems.
Limitations
- Only five Spider databases and 90 questions, limiting coverage.
- Verbalized text uses templates, which may not reflect natural document style.
- LLM-based judge (gpt-4o-mini) with 93.8% agreement on only 20 questions.
Experimental setup
Datasets: Spider
Benchmarks: HoloBench
Metrics: recall-oriented accuracy (exact/partial/no match scoring) unweighted average accuracy context-length-weighted average accuracy
Baselines: Needle-in-a-haystack BABILong FLenQA bge-large-en-v1.5 RAG
Builds on: Spider Neural Databases Needle-in-a-haystack RAG
Closest papers by meaning
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.89)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.88)
- TableRAG: Million-Token Table Understanding with Language Models (NeurIPS 2024 · similarity 0.87)
- Sufficient Context: A New Lens on Retrieval Augmented Generation Systems (ICLR 2025 · similarity 0.87)
- Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model (ICLR 2025 · similarity 0.87)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.