Research map / Large language models / Language, safety and interpretability
Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces needle-threading and multi-threading retrieval tasks on synthetic UUID key-value haystacks to evaluate 17 LLMs up to ~630k LLaMA-3.1 tokens, and proposes a tokenizer-independent effective context length metric.
Problem
Existing long-context benchmarks saturate on simple needle retrieval, are limited to sub-100k contexts, and give only coarse macro-trends, so it is unclear how effectively frontier LLMs use their advertised context windows, especially for multi-step threads.
Approach
Synthetic UUID key-value haystacks where values chain into keys form threads; single, multiple, conditional, threading, multi-threading and branched variants isolate granular factors (context length, thread direction, number of threads). A contour-based effective context length is computed in characters to avoid tokenizer differences.
Key results
- Gemini 1.5 Pro scores 87.7% at 1.2k and 65.2% at 128k on the aggregate score; GPT-4o is best at the smallest contexts (93.2% at 1.2k).
- Effective context length is far below advertised limits: e.g., GPT-4o's threading effective length is 7k characters (3% of its 214k-character limit), while Gemini 1.5 Pro's single-needle effective length is 315k characters (13%).
- Claude 3.5 Sonnet shows no degradation in multi-threading up to 25 threads, supporting that leading LLMs are thread-safe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase haystack context length | LLMs on Single Needle retrieval · Single Needle | retrieval accuracy | Hurt |
| Increase haystack context length | LLMs on Multiple Needles retrieval · Multiple Needles | retrieval accuracy | Hurt |
| Cluster relevant key-value pairs rather than randomly place them | Conditional Needles retrieval · Conditional Needles | retrieval accuracy | Helped |
| Cluster specified key-value pairs rather than randomly place them | Multiple Needles retrieval · Multiple Needles | retrieval accuracy | No clear effect |
| Increase the number of concurrently retrieved needles | stronger models on Multiple Needles retrieval · Multiple Needles | retrieval accuracy | No clear effect |
| Increase the number of condition-matching needles in random positions | Conditional Needles retrieval · Conditional Needles | retrieval accuracy | Hurt |
| Move thread links forward through the context rather than backward | LLMs on Threading · Threading | retrieval accuracy | Helped |
| Increase thread length | LLMs on Threading · Threading | retrieval accuracy | Hurt |
| Follow additional threads simultaneously | LLMs on synthetic Multi-threading · Multi-threading | retrieval accuracy | No clear effect |
| Use natural-language text instead of synthetic UUID haystacks | LLMs on Multi-threading · Natural-language Multi-threading | retrieval accuracy | Hurt |
| Increase branching factor | LLMs on Branched Threading · Branched Threading | retrieval accuracy | Mixed |
Surprising findings
- Most models perform far worse at following threads backward than forward through the context, and Claude 3.5 Sonnet is an exception.
- Tokenizers differ greatly: a UUID pair is about 50 GPT-4o tokens versus 75 Gemini 1.5 tokens, so a 1M Gemini limit is about 700k GPT-4o tokens.
Limitations
- Only synthetic UUID data; does not capture domain-specific behaviour.
- Limited number of repeats for expensive models due to API cost and rate limits.
- Some models (e.g., Mistral) could not be evaluated near their context limits due to API restrictions.
Experimental setup
Datasets: synthetic UUID key-value haystacks Gibbon's The History of the Decline and Fall of the Roman Empire (natural language ablation)
Benchmarks: Single Needle Multiple Needles Conditional Needles Threading Multi-threading Branched Threading natural language threading ablation
Metrics: exact match accuracy effective context length (contour at 75% threshold, in chara
Baselines: GPT-4o Gemini 1.5 Pro Gemini 1.5 Flash Claude 3.5 Sonnet LLaMA 3.1 405B
Builds on: needle in a haystack
Closest papers by meaning
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.90)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.90)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.90)
- Structured Packing in LLM Training Improves Long Context Utilization (AAAI 2025 · similarity 0.89)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.89)
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.