Research map / Large language models / Language, safety and interpretability
Training-Free Long-Context Scaling of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dual Chunk Attention (DCA), a training-free method that remaps RoPE position indices via intra-chunk, inter-chunk and successive-chunk attention to extend Llama2/Llama3 and other RoPE models' context windows to 100k+ tokens, compatible with Flash Attention.
Problem
LLMs degrade sharply when input length exceeds pretraining context, and existing fixes (PI, NTK scaling) either need costly finetuning or increase perplexity at lengths beyond twice the training length in training-free settings.
Approach
Segment long sequences into chunks smaller than the pretraining window and reassign RoPE position indices so that intra-chunk, adjacent-chunk (successive) and distant-chunk (inter) query-key relative positions all stay within the pretrained range, with no weight updates and Flash Attention compatibility.
Key results
- ChunkLlama2 70B keeps PG19 perplexity between 5.18 and 5.59 across 4k to 96k context, while Llama2 70B and PI/NTK baselines blow up beyond 16k.
- Training-free ChunkLlama2 70B-Chat reaches 94% of gpt-3.5-turbo-16k performance on four L-Eval tasks and averages 37.8 vs LongLoRA 70B's 37.2 on the few-shot long-document benchmarks.
- DCA applied to CodeLlama and Together-32k extends their usable context to 192k tokens with around 90% passkey retrieval accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Dual Chunk Attention | Llama2 7B · PG19 validation, 32768 tokens | perplexity: >10^2 -> 7.89 | Helped |
| Apply Dual Chunk Attention | Llama2 70B · PG19 validation, 32768 tokens | perplexity: >10^2 -> 5.30 | Helped |
| Apply Dual Chunk Attention | CodeLlama 7B · PG19 validation, 192k tokens | perplexity: 24.78 -> 9.83 | Helped |
| Apply Dual Chunk Attention | Together 7B · PG19 validation, 192k tokens | perplexity: >10^2 -> 7.83 | Helped |
| Apply Dual Chunk Attention | Llama2 70B · Four research benchmarks, average | average score: 29.5 -> 37.8 | Helped |
| Use intra-chunk attention only, omitting information from previous chunks (ablation) | Dual Chunk Attention · Passkey retrieval | retrieval accuracy | Hurt |
| Add inter-chunk attention to intra-chunk attention (ablation) | Dual Chunk Attention · Passkey retrieval, 12k input | retrieval performance | Helped |
| Add inter-chunk attention without successive-chunk attention (ablation) | Dual Chunk Attention · Language modeling | perplexity | Hurt |
| Add successive-chunk attention to intra-chunk and inter-chunk attention (ablation) | Dual Chunk Attention · Language modeling and passkey retrieval | perplexity and retrieval accuracy | Helped |
| Select the closest in-context examples instead of random examples (ablation) | ChunkLlama2 7B/13B · Qasper and QuALITY | F1 and exact match: 28.2 -> 27.3; 35.6 -> 33.9; 29.3 -> 28.5; 47.9 -> 46.2 | Hurt |
Surprising findings
- Using the most similar in-context examples (Example Best) gives the poorest QA results, since models copy the example's answer.
- On the Llama2 70B passkey test, the first accuracy drop occurs at the beginning of the text rather than the middle, unlike PI and NTK.
Limitations
- Perplexity degrades beyond about 128k tokens for Llama2 70B (5.30 at 32k to 7.05 at 192k) and fails for 7B at 96k+
- Global understanding questions on the paper still fail, and 13B answers remain imprecise
Experimental setup
Datasets: PG19 ShareGPT AlpacaGPT4 NarrativeQA Qasper QuALITY QMSum L-Eval
Benchmarks: PG19 passkey retrieval NarrativeQA Qasper QuALITY QMSum L-Eval
Metrics: perplexity retrieval accuracy F1 exact match ROUGE-geometric mean
Baselines: Llama2 Long LongLoRA Together-32k CodeLlama YaRN CLEX ReRoPE GPT-3.5-16k
Builds on: RoPE Position Interpolation NTK-Aware RoPE Flash Attention 2 LongChat
Closest papers by meaning
- Efficient Long Context Fine-tuning with Chunk Flow (ICML 2025 · similarity 0.91)
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · similarity 0.90)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.89)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.89)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.