Research map / Large language models / Language, safety and interpretability
NoLiMa: Long-Context Evaluation Beyond Literal Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NoLiMa, a needle-in-a-haystack benchmark whose question-needle pairs have minimal lexical overlap, requiring latent associative (one- and two-hop) reasoning, and evaluates 13+ long-context LLMs showing sharp degradation with context length.
Problem
Existing needle-in-a-haystack and long-context benchmarks contain literal matches between questions and relevant context, letting models exploit surface cues and overstating long-context ability.
Approach
Design needles and questions with minimal lexical overlap linked only by latent associations (world knowledge or commonsense, one or two hops), embedded in book haystacks filtered for distractors and false answers, so that surface matching has no utility.
Key results
- At 32K, 11 of 13 evaluated models fall to half or less of their base scores; GPT-4o drops from 99.3% base to 69.7% at 32K.
- Llama 3.1 70B has effective length of only 2K on NoLiMa versus 16K on BABILong and 32K on RULER.
- Literal-match multiple-choice format raises Llama 3.3 70B two-hop 32K accuracy from 25.9% to 87.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase context length from the short-context baseline to 32K | NoLiMa models · NoLiMa 32K | accuracy: GPT-4o: 99.3 -> 69.7; 11 models drop below 50% of their base score | Hurt |
| Add CoT prompting (ablation) | Llama 3.3 70B · NoLiMa one-hop 4K | accuracy: 90.3 -> 95.6 | Helped |
| Add CoT prompting (ablation) | Llama 3.3 70B · NoLiMa one-hop 32K | accuracy: 56.2 -> 60.6 | Helped |
| Add CoT prompting (ablation) | Llama 3.3 70B · NoLiMa two-hop 4K | accuracy: 70.7 -> 82.4 | Helped |
| Add CoT prompting (ablation) | Llama 3.3 70B · NoLiMa two-hop 32K | accuracy: 25.9 -> 34.3 | Helped |
| Add a literal match by asking directly about the needle fact (ablation) | Llama 3.3 70B · NoLiMa direct 32K | accuracy: + | Helped |
| Add literal-match multiple-choice answer options (ablation) | Llama 3.3 70B · NoLiMa one-hop 32K | accuracy: 56.2 -> 93.1 | Helped |
| Add literal-match multiple-choice answer options (ablation) | Llama 3.3 70B · NoLiMa two-hop 32K | accuracy: 25.9 -> 87.2 | Helped |
| Add an irrelevant sentence containing the query keyword (ablation) | GPT-4o and Llama 3.3 70B · NoLiMa with distractor | length generalization | Hurt |
| Increase associative reasoning from one hop to two hops (ablation) | NoLiMa models · NoLiMa long contexts | accuracy | Hurt |
| Invert the order of character and needle keyword in the fact (ablation) | NoLiMa models · NoLiMa | accuracy | Hurt |
| Use reasoning-based models instead of standard CoT prompting | NoLiMa-Hard · 32K | accuracy | Mixed |
Surprising findings
- Even reasoning models (GPT-o1, DeepSeek R1-Distill-Llama-70B) fall below 50% of base score at 32K on the hard subset.
- Distractor sentences containing the query keyword drop GPT-4o's effective length to 1K.
Limitations
- Needle set is small (58 question-needle pairs, 28 keyword pairs), and haystacks come from 10 books.
- Haystack filtering relies on a semi-automatic process with manual review by one author and Llama 3.3 70B flagging.
- Mechanistic explanation of attention failures is not analyzed in depth and is left to future work.
Experimental setup
Datasets: open-license books (10 books, each at least 50K tokens) as haystack source
Benchmarks: NoLiMa NoLiMa-Hard
Metrics: accuracy base score effective length (85% of base score threshold) normalized score
Baselines: GPT-4o Gemini 1.5 Pro Llama 3.3 70B Llama 3.1 405B Claude 3.5 Sonnet GPT-o1 GPT-o3 Mini DeepSeek R1-Distill-Llama-70B
Builds on: Needle-in-a-Haystack RULER BABILong Contriever
Closest papers by meaning
- A Controllable Examination for Long-Context Language Models (NeurIPS 2025 · similarity 0.91)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.90)
- Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks? (ICLR 2025 · similarity 0.89)
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly (ICLR 2025 · similarity 0.88)
- Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack (NeurIPS 2024 · similarity 0.88)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.