Research map / Large language models / Language, safety and interpretability

Short-Context Dominance: How Much Local Context Natural Language Actually Needs?

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that for most long-document sequences a 32-96 token local prefix suffices for LLM next-token prediction, proposes DaMCL/LSDS to detect long-context sequences without ground truth, and TaBoo decoding that boosts long-context-relevant tokens for improved QA.

Problem

How much local context do language models actually need to predict the next token, and can long-context-requiring sequences be detected and their generation improved without ground-truth next tokens?

Approach

Defines minimal context length (MCL) using LLMs as oracles, a distribution-based variant (DaMCL) using JSD between short-prefix and full-context next-token distributions, and a detector (LSDS) plus token-level LSPS shift to drive targeted boosting during decoding (TaBoo).

Key results

• For 1-7k token sequences from long-context documents, 75-80% need at most the last 96 tokens for MCL.
• LSDS with threshold 0.6 classifies 80-90% of oracle-labeled long-context sequences correctly while mislabeling fewer than 5-10% of short ones.
• TaBoo outperforms vanilla nucleus sampling across all model-dataset pairs and beats CAD on F1 in 11 of 12 combinations.

Surprising findings

• Even on inherently long-context datasets like GovReport, MCL distributions remain heavily skewed toward very short prefixes.
• Stricter JSD thresholds (0.1) produce a U-shaped DaMCL distribution, with sequences either resolved by short context or requiring nearly the full input.

Limitations

• Results depend on the confidence threshold and oracle choice; MCL relies on ground-truth next tokens.
• Evaluation of TaBoo limited to three QA datasets and 7-8B models with F1/BLEU/ROUGE metrics.

Experimental setup

Datasets: WritingPrompts CNN/DailyMail WikiText-103 GovReport QMSum BookSum NarrativeQA HotpotQA MultiFieldQA-en Wikimedia CCDV PubMed OpenWebMath

Benchmarks: NarrativeQA HotpotQA MultiFieldQA-en

Metrics: F1 BLEU ROUGE-L AUC-like classification agreement with oracles power-law exponent

Baselines: Vanilla nucleus sampling CAD

Builds on: context-aware decoding nucleus sampling needle-in-a-haystack

Closest papers by meaning

• Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.89)
• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.88)
• L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling (NeurIPS 2025 · similarity 0.88)
• Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.88)
• Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.87)
• Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization (ICLR 2025 · similarity 0.87)