Research map / Large language models / Retrieval and knowledge
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BABILong, a scalable generative benchmark that hides bAbI reasoning facts inside PG19 book text to test long-context reasoning up to 10M tokens (50M for ARMT), and evaluates LLMs, RAG, long-context methods and fine-tuned recurrent memory models.
Problem
Existing long-context benchmarks (e.g., needle-in-a-haystack) are too simple, short, or unable to test multi-fact reasoning over very long contexts, so the effective context utilization of LLMs is unclear.
Approach
Generate arbitrarily long inputs by interleaving bAbI reasoning sentences (20 tasks) with natural PG19 background text, so models must find, store and reason over distributed facts; evaluate a wide range of LLMs, RAG pipelines, and fine-tuned recurrent memory transformers (RMT, ARMT, Mamba) on it.
Key results
- Popular LLMs use only about 10-20% of their claimed context; most models stay above 85% on QA1 only up to about 4K tokens.
- RAG achieves about 60% accuracy on single-fact QA1 regardless of length but fails on multi-fact QA2/QA3, falling below random guessing.
- Fine-tuned RMT (137M) maintains high accuracy up to 10M tokens and ARMT reaches 50M tokens, far beyond its 16K training length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sentence-based retrieval instead of 512-token chunk retrieval (ablation) | RAG on BABILong QA1 · BABILong QA1 | accuracy | Helped |
| Increase retrieved sentences from top-5 to top-20 (ablation) | Sentence-based RAG on BABILong QA1 · BABILong QA1 | accuracy | No clear effect |
| Fine-tune GPT-3.5-Turbo on QA1 | GPT-3.5-Turbo · BABILong QA1 across context lengths | accuracy | Helped |
| Fine-tune Mistral-7B on QA1 | Mistral-7B · BABILong QA1 across context lengths | accuracy | Helped |
| Fine-tune GPT-3.5 on QA1 (ablation) | GPT-3.5 · BABILong QA2-QA5, no distractor text | accuracy | Helped |
| Fine-tune Mistral-7B on QA1 (ablation) | Mistral-7B · BABILong QA2-QA5, no distractor text | accuracy | Hurt |
| Use retrieval-augmented generation instead of direct prompting (ablation) | GPT-4 and Llama-3 · BABILong long-context tasks | accuracy | No clear effect |
| Use RAG for multi-fact retrieval | BABILong QA2 and QA3 · BABILong QA2 and QA3 | accuracy | Hurt |
| Fine-tune recurrent memory models on BABILong | RMT and ARMT · BABILong, up to 10M tokens | accuracy | Helped |
| Scale recurrent memory models beyond training sequence length | RMT · BABILong, up to 11.1M tokens | accuracy | No clear effect |
| Use ARMT for very long sequences | ARMT · BABILong, up to 50M tokens | sequence length processed: 50 million tokens | Helped |
| Use Mamba instead of recurrent memory models | Fine-tuned models on BABILong QA3 · BABILong QA3 | accuracy | Helped |
Surprising findings
- RAG fails on multi-hop tasks QA2 and QA3, with accuracy dropping below random guessing.
- Small fine-tuned recurrent models (137M) outperform much larger LLMs such as GPT-4 on long-context tasks.
Limitations
- Only PG19 and Wiki were tried as background text sources; other distributions may change results.
- The fact vocabulary reused from bAbI is limited, making tasks easier for fine-tuned models.
- RAG and prompt choices were not optimized, so the RAG results may be suboptimal.
Experimental setup
Datasets: bAbI PG19
Benchmarks: BABILong MMLU RULER
Metrics: accuracy correlation with MMLU
Baselines: GPT-4 Gemini 1.5 Pro GPT-3.5-Turbo Mamba RAG-C RAG-S Yarn Activation Beacon RULER MMLU
Builds on: bAbI PG19 Recurrent Memory Transformer Associative RMT RULER
Closest papers by meaning
- A Controllable Examination for Long-Context Language Models (NeurIPS 2025 · similarity 0.90)
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML 2025 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.89)
- Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data (ICLR 2025 · similarity 0.89)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.