Research map / Large language models / Language, safety and interpretability
A Controllable Examination for Long-Context Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LongBioBench, a synthetic long-context benchmark built from artificial biographies that are coherent with the haystack, configurable, and scored by exact match, and uses it to evaluate 18 LCLMs across understanding, reasoning, and trustworthiness.
Problem
Existing long-context benchmarks are either realistic but costly, contaminated and hard to interpret, or synthetic (needle-in-a-haystack) with incoherent needles and numerical needles that create shortcuts, so they poorly reflect real capabilities.
Approach
Embed a fluent, semantically coherent needle biography within a haystack of artificially generated biographies sampled from attribute pools, and extend the base retrieval task with configurable reasoning (calculation, ranking, two-diff, multi-hop) and trustworthiness (citation, refusal) subtasks scored by exact match rather than LLM-as-judge.
Key results
- Across 18 LCLMs, GPT-4o, Qwen2.5-14B-1M and Qwen2.5-7B-1M exceed 85% on understanding, but the best reasoning accuracy is only 66.5% and no model exceeds 90% on trustworthiness tasks.
- LongBioBench scores correlate with HELMET at 0.853 versus 0.559 for RULER.
- Replacing coherent bios with a Paul Graham essay haystack (BiaH) inflates scores, with gaps of -7.9% on standard retrieval growing to -88.9% on harder tasks for Qwen2.5-7B-1M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace coherent biography context with an incoherent Paul Graham essay haystack (BiaH) (ablation) | Qwen2.5-Instruct-7B-1M · LongBioBench tasks, compared with BiaH | accuracy: -7.9% | Hurt |
| replace coherent biography context with an incoherent Paul Graham essay haystack (BiaH) (ablation) | Qwen2.5-Instruct-7B-1M · LongBioBench harder retrieval scenarios | accuracy: -28.3% | Hurt |
| replace coherent biography context with an incoherent Paul Graham essay haystack (BiaH) (ablation) | Qwen2.5-Instruct-7B-1M · LongBioBench most complex retrieval scenarios | accuracy: -88.9% | Hurt |
| increase context length from 2k to 8k | Qwen2.5-14B-Instruct-1M · LongBioBench ICL | accuracy: 51.5 -> 25.5 | Hurt |
| increase context length from 8k to 16k | Qwen2.5-7B-Instruct-1M · LongBioBench ICL | accuracy: 19.5 -> 11.0 | Hurt |
| extend context from 256k to 512k | Qwen2.5-7B-1M · LongBioBench understanding | accuracy: 48.0 -> 0.3 | Hurt |
| extend context from 256k to 512k | Qwen2.5-14B-1M · LongBioBench understanding | accuracy: 47.0 -> 2.7 | Hurt |
| extend long-context continual pretraining beyond the early stage | Qwen2.5-7B · LongBioBench retrieval at 32k | accuracy | Mixed |
| extend long-context continual pretraining | Qwen2.5-7B · LongBioBench reasoning at 32k | accuracy: ~10% | No clear effect |
| extend long-context continual pretraining | Qwen2.5-7B · LongBioBench trustworthiness at 32k | accuracy | Hurt |
| increase distractor density (ablation) | Qwen2.5-Instruct-7B-1M · LongBioBench calculation and pronoun retrieval | accuracy | Hurt |
Surprising findings
- Long-context continual pretraining of Qwen2.5-7B saturates early (retrieval peaks around 4K steps) and trustworthiness declines as training proceeds.
- Some models score higher on the calculation task than on the 2-retrieval task, suggesting stronger sensitivity to numerical than textual information.
Limitations
- Only the simplest extension of each task is studied; closed-source models like Gemini and Claude are not evaluated due to budget.
- Not a perfect proxy for real-world tasks, with weak correlation on rank and twodiff tasks versus HELMET.
Experimental setup
Datasets: Kaggle attribute lists (universities, cities) LongBioBench generated biographies
Benchmarks: LongBioBench HELMET
Metrics: exact-match accuracy all-or-nothing accuracy citation accuracy IDK accuracy Spearman correlation
Baselines: RULER HELMET BABILong NoLiMa Michelangelo OpenAI-MRCR
Builds on: Needle-in-a-Haystack bioS dataset RULER HELMET
Closest papers by meaning
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly (ICLR 2025 · similarity 0.91)
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML 2025 · similarity 0.91)
- What is Wrong with Perplexity for Long-context Language Modeling? (ICLR 2025 · similarity 0.91)
- Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack (NeurIPS 2024 · similarity 0.90)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.90)
- Understanding Synthetic Context Extension via Retrieval Heads (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.